I'm writing this to drive some forward motion and to nerd-snipe those
who know better than I do into putting their thoughts into words.
There are three ways to process JSON[1]
- Streaming (Push or Pull)
- Traversing a Tree (Realized or Lazy)
- Declarative Databind (N ways)
Of these, JEP-198 explicitly ruled out providing "JAXB style type safe
data binding."
No justification is given, but if I had to insert my own: mapping the
Json model to/from the Java/JVM object model is a cursed combo of
- Huge possible design space
- Unpalatably large surface for backwards compatibility
- Serialization! Boo![2]
So for an artifact like the JDK, it probably doesn't make sense to
include. That tracks.
It won't make everyone happy, people like databind APIs, but it tracks.
So for the "read flow" these are the things to figure out.
| Should Provide? | Intended User(s) |
----------------+-----------------+------------------+
Streaming Push | | |
----------------+-----------------+------------------+
Streaming Pull | | |
----------------+-----------------+------------------+
Realized Tree | | |
----------------+-----------------+------------------+
Lazy Tree | | |
----------------+-----------------+------------------+
At which point, we should talk about what "meets needs of Java
developers using JSON" implies.
JSON is ubiquitous. Most kinds of software us schmucks write could
have a reason to interact with it.
The full set of "user personas" therefore aren't practical for me to
talk about.[3]
JSON documents, however, are not so varied.
- There are small ones (1-10kb)
- There are medium ones (10-1000kb)
- There are big ones (1000kb-???)
- There are shallow ones
- There are deep ones
So that feels like an easier direction to talk about it from.
This repo[4] has some convenient toy examples of how some of those
APIs look in libraries
in the ecosystem. Specifically the Streaming Pull and Realized Tree
models.
User r = new User();
while (true) {
JsonToken token = reader.peek();
switch (token) {
case BEGIN_OBJECT:
reader.beginObject();
break;
case END_OBJECT:
reader.endObject();
return r;
case NAME:
String fieldname = reader.nextName();
switch (fieldname) {
case "id":
r.setId(reader.nextString());
break;
case "index":
r.setIndex(reader.nextInt());
break;
...
case "friends":
r.setFriends(new ArrayList<>());
Friend f = null;
carryOn = true;
while (carryOn) {
token = reader.peek();
switch (token) {
case BEGIN_ARRAY:
reader.beginArray();
break;
case END_ARRAY:
reader.endArray();
carryOn = false;
break;
case BEGIN_OBJECT:
reader.beginObject();
f = new Friend();
break;
case END_OBJECT:
reader.endObject();
r.getFriends().add(f);
break;
case NAME:
String fn = reader.nextName();
switch (fn) {
case "id":
f.setId(reader.nextString());
break;
case "name":
f.setName(reader.nextString());
break;
}
break;
}
}
break;
}
}
I think its not hard to argue that the streaming apis are brutalist.
The above is Gson, but Jackson, moshi, etc
seem at least morally equivalent.
Its hard to write, hard to write *correctly*, and theres is a curious
protensity towards pairing it
with anemic, mutable models.
That being said, it handles big documents and deep documents really
well. It also performs
pretty darn well and is good enough as a "fallback" when the intended
user experience
is through something like databind.
So what could we do meaningfully better with the language we have
today/will have tommorow?
- Sealed interfaces + Pattern matching could give a nicer model for tokens
sealed interface JsonToken {
record Field(String name) implements JsonToken {}
record BeginArray() implements JsonToken {}
record EndArray() implements JsonToken {}
record BeginObject() implements JsonToken {}
record EndObject() implements JsonToken {}
// ...
}
// ...
User r = new User();
while (true) {
JsonToken token = reader.peek();
switch (token) {
case BeginObject __:
reader.beginObject();
break;
case EndObject __:
reader.endObject();
return r;
case Field("id"):
r.setId(reader.nextString());
break;
case Field("index"):
r.setIndex(reader.nextInt());
break;
// ...
case Field("friends"):
r.setFriends(new ArrayList<>());
Friend f = null;
carryOn = true;
while (carryOn) {
token = reader.peek();
switch (token) {
// ...
- Value classes can make it all more efficient
sealed interface JsonToken {
value record Field(String name) implements JsonToken {}
value record BeginArray() implements JsonToken {}
value record EndArray() implements JsonToken {}
value record BeginObject() implements JsonToken {}
value record EndObject() implements JsonToken {}
// ...
}
- (Fun One) We can transform a simpler-to-write push parser into a
pull parser with Coroutines
This is just a toy we could play with while making something in
the JDK. I'm pretty sure
we could make a parser which feeds into something like
interface Listener {
void onObjectStart();
void onObjectEnd();
void onArrayStart();
void onArrayEnd();
void onField(String name);
// ...
}
and invert a loop like
while (true) {
char c = next();
switch (c) {
case '{':
listener.onObjectStart();
// ...
// ...
}
}
by putting a Coroutine.yield in the callback.
That might be a meaningful simplification in code structure, I
don't know enough to say.
But, I think there are some hard questions like
- Is the intent[5] to be make backing parser for ecosystem databind apis?
- Is the intent that users who want to handle big/deep documents fall
back to this?
- Are those new language features / conveniences enough to offset the
cost of committing to a new api?
- To whom exactly does a low level api provide value?
- What benefit is standardization in the JDK?
and just generally - who would be the consumer(s) of this?
The other kind of API still on the table is a Tree. There are two ways
to handle this
1. Load it into `Object`. Use a bunch of instanceof checks/casts to
confirm what it actually is.
Object v;
User u = new User();
if ((v = jso.get("id")) != null) {
u.setId((String) v);
}
if ((v = jso.get("index")) != null) {
u.setIndex(((Long) v).intValue());
}
if ((v = jso.get("guid")) != null) {
u.setGuid((String) v);
}
if ((v = jso.get("isActive")) != null) {
u.setIsActive(((Boolean) v));
}
if ((v = jso.get("balance")) != null) {
u.setBalance((String) v);
}
// ...
if ((v = jso.get("latitude")) != null) {
u.setLatitude(v instanceof BigDecimal ? ((BigDecimal)
v).doubleValue() : (Double) v);
}
if ((v = jso.get("longitude")) != null) {
u.setLongitude(v instanceof BigDecimal ? ((BigDecimal)
v).doubleValue() : (Double) v);
}
if ((v = jso.get("greeting")) != null) {
u.setGreeting((String) v);
}
if ((v = jso.get("favoriteFruit")) != null) {
u.setFavoriteFruit((String) v);
}
if ((v = jso.get("tags")) != null) {
List<Object> jsonarr = (List<Object>) v;
u.setTags(new ArrayList<>());
for (Object vi : jsonarr) {
u.getTags().add((String) vi);
}
}
if ((v = jso.get("friends")) != null) {
List<Object> jsonarr = (List<Object>) v;
u.setFriends(new ArrayList<>());
for (Object vi : jsonarr) {
Map<String, Object> jso0 = (Map<String, Object>) vi;
Friend f = new Friend();
f.setId((String) jso0.get("id"));
f.setName((String) jso0.get("name"));
u.getFriends().add(f);
}
}
2. Have an explicit model for Json, and helper methods that do said
casts[6]
this.setSiteSetting(readFromJson(jsonObject.getJsonObject("site")));
JsonArray groups = jsonObject.getJsonArray("group");
if(groups != null)
{
int len = groups.size();
for(int i=0; i<len; i++)
{
JsonObject grp = groups.getJsonObject(i);
SNMPSetting grpSetting = readFromJson(grp);
String grpName = grp.getString("dbgroup", null);
if(grpName != null && grpSetting != null)
this.groupSettings.put(grpName, grpSetting);
}
}
JsonArray hosts = jsonObject.getJsonArray("host");
if(hosts != null)
{
int len = hosts.size();
for(int i=0; i<len; i++)
{
JsonObject host = hosts.getJsonObject(i);
SNMPSetting hostSetting = readFromJson(host);
String hostName = host.getString("dbhost", null);
if(hostName != null && hostSetting != null)
this.hostSettings.put(hostName, hostSetting);
}
}
I think what has become easier to represent in the language nowadays
is that explicit model for Json.
Its the 101 lesson of sealed interfaces.[7] It feels nice and clean.
sealed interface Json {
final class Null implements Json {}
final class True implements Json {}
final class False implements Json {}
final class Array implements Json {}
final class Object implements Json {}
final class String implements Json {}
final class Number implements Json {}
}
And the cast-and-check approach is now more viable on account of
pattern matching.
if (jso.get("id") instanceof String v) {
u.setId(v);
}
if (jso.get("index") instanceof Long v) {
u.setIndex(v.intValue());
}
if (jso.get("guid") instanceof String v) {
u.setGuid(v);
}
// or
if (jso.get("id") instanceof String id &&
jso.get("index") instanceof Long index &&
jso.get("guid") instanceof String guid) {
return new User(id, index, guid, ...); // look ma, no setters!
}
And on the horizon, again, is value types.
But there are problems with this approach beyond the performance
implications of loading into
a tree.
For one, all the code samples above have different behaviors around
null keys and missing keys
that are not obvious from first glance.
This won't accept any null or missing fields
if (jso.get("id") instanceof String id &&
jso.get("index") instanceof Long index &&
jso.get("guid") instanceof String guid) {
return new User(id, index, guid, ...);
}
This will accept individual null or missing fields, but also will
silently ignore
fields with incorrect types
if (jso.get("id") instanceof String v) {
u.setId(v);
}
if (jso.get("index") instanceof Long v) {
u.setIndex(v.intValue());
}
if (jso.get("guid") instanceof String v) {
u.setGuid(v);
}
And, compared to databind where there is information about the
expected structure of the document
and its the job of the framework to assert that, I posit that the
errors that would be encountered
when writing code against this would be more like
"something wrong with user"
than
"problem at users[5].name, expected string or null. got 5"
Which feels unideal.
One approach I find promising is something close to what Elm does with
its decoders[8]. Not just combining assertion
and binding like what pattern matching with records allows, but
including a scheme for bubbling/nesting errors.
static String string(Json json) throws JsonDecodingException {
if (!(json instanceof Json.String jsonString)) {
throw JsonDecodingException.of(
"expected a string",
json
);
} else {
return jsonString.value();
}
}
static <T> T field(Json json, String fieldName, Decoder<? extends
T> valueDecoder) throws JsonDecodingException {
var jsonObject = object(json);
var value = jsonObject.get(fieldName);
if (value == null) {
throw JsonDecodingException.atField(
fieldName,
JsonDecodingException.of(
"no value for field",
json
)
);
}
else {
try {
return valueDecoder.decode(value);
} catch (JsonDecodingException e) {
throw JsonDecodingException.atField(
fieldName,
e
);
} catch (Exception e) {
throw JsonDecodingException.atField(fieldName,
JsonDecodingException.of(e, value));
}
}
}
Which I think has some benefits over the ways I've seen of working
with trees.
- It is declarative enough that folks who prefer databind might be
happy enough.
static User fromJson(Json json) {
return new User(
Decoder.field(json, "id", Decoder::string),
Decoder.field(json, "index", Decoder::long_),
Decoder.field(json, "guid", Decoder::string),
);
}
/ ...
List<User> users = Decoders.array(json, User::fromJson);
- Handling null and optional fields could be less easily conflated
Decoder.field(json, "id", Decoder::string);
Decoder.nullableField(json, "id", Decoder::string);
Decoder.optionalField(json, "id", Decoder::string);
Decoder.optionalNullableField(json, "id", Decoder::string);
- It composes well with user defined classes
record Guid(String value) {
Guid {
// some assertions on the structure of value
}
}
Decoder.string(json, "guid", guid -> new Guid(Decoder.string(guid)));
// or even
record Guid(String value) {
Guid {
// some assertions on the structure of value
}
static Guid fromJson(Json json) {
return new Guid(Decoder.string(guid));
}
}
Decoder.string(json, "guid", Guid::fromJson);
- When something goes wrong, the API can handle the fiddlyness of
capturing information for feedback.
In the code I've sketched out its just what field/index things
went wrong at. Potentially
capturing metadata like row/col numbers of the source would be
sensible too.
Its just not reasonable to expect devs to do extra work to get
that and its really nice to give it.
There are also some downsides like
- I do not know how compatible it would be with lazy trees.
Lazy trees being the only way that a tree api could handle big or
deep documents.
The general concept as applied in libraries like json-tree[9] is
to navigate without
doing any work, and that clashes with wanting to instanceof check
the info at the
current path.
- It *almost* gives enough information to be a general schema approach
If one field fails, that in the model throws an exception
immediately. If an API should
return "errors": [...], that is inconvenient to construct.
- None of the existing popular libraries are doing this
The only mechanics that are strictly required to give this sort
of API is lambdas. Those have
been out for a decade. Yes sealed interfaces make the data model
prettier but in concept you
can build the same thing on top of anything.
I could argue that this is because of "cultural momentum" of
databind or some other reason,
but the fact remains that it isn't a proven out approach.
Writing Json libraries is a todo list[10]. There are a lot of bad
ideas and this might be one of the,
- Performance impact of so many instanceof checks
I've gotten a 4.2% slowdown compared to the "regular" tree code
without the repeated casts.
But that was with a parser that is 5x slower than Jacksons. (using
the same benchmark project as for the snippets).
I think there could be reason to believe that the JIT does well
enough with repeated instanceof
checks to consider it.
My current thinking is that - despite not solving for large or deep
documents - starting with a really "dumb" realized tree api
might be the right place to start for the read side of a potential
incubator module.
But regardless - this feels like a good time to start more concrete
conversations. I fell I should cap this email since I've reached the
point of decoherence and haven't even mentioned the write side of things
[1]: http://www.cowtowncoder.com/blog/archives/2009/01/entry_131.html
[2]: https://security.snyk.io/vuln/maven?search=jackson-databind
[3]: I only know like 8 people
[4]:
https://github.com/fabienrenaud/java-json-benchmark/blob/master/src/main/java/com/github/fabienrenaud/jjb/stream/UsersStreamDeserializer.java
[5]: When I say "intent", I do so knowing full well no one has been
actively thinking of this for an entire Game of Thrones
[6]:
https://github.com/yahoo/mysql_perf_analyzer/blob/master/myperf/src/main/java/com/yahoo/dba/perf/myperf/common/SNMPSettings.java
[7]: https://www.infoq.com/articles/data-oriented-programming-java/
[8]: https://package.elm-lang.org/packages/elm/json/latest/Json-Decode
[9]: https://github.com/jbee/json-tree
[10]: https://stackoverflow.com/a/14442630/2948173
[11]: In 30 days JEP-198 it will be recognizably PI days old for the
2nd time in its history.
[12]: To me, the fact that is still an open JEP is more a social
convenience than anything. I could just as easily writing this exact
same email about TOML.
