> The language here could be similar...implementations are not required to serialize them any differently but are allowed to produce nicer looking JSON for extension types when they can guarantee it's valid (many implementations can do this, at least for canonical extensions whose serialization has been voted on and is part of the spec, sort of).
The problem i have with this is just that it means we either have to complicate how we define equality or the JSON representation can't round trip without significant special casing for implementations with extension types which we're trying to avoid. I just don't think the convenience of nicer looking JSON is worth the complexity. i.e. I agree with what David said earlier in this thread. > All that said, just making "metadata" an object instead of a list of key values leads to pretty equivalent extraction of an extension parameter We discussed this a bit and landed on using the list of key values to be more like the IPC representation, but I wouldn't object to making metadata an object. --Matt On Fri, Jul 10, 2026, 10:47 PM Dewey Dunnington <[email protected]> wrote: > > To practically implement inline JSON for extension types, though > > The way we do this in GeoArrow (Arrow Go, Arrow C++ Parquet, > arrow-rs/parquet) to prevent escaped-json-in-json when going from Parquet > (crs is a string) -> GeoArrow (crs is a member of a JSON object) is to > parse the value, and if it parses as a JSON object, use that instead of the > stringified version. The language we have in the spec for this is > "Producers should not write an escaped JSON object to the crs key" [1]. The > language here could be similar...implementations are not required to > serialize them any differently but are allowed to produce nicer looking > JSON for extension types when they can guarantee it's valid (many > implementations can do this, at least for canonical extensions whose > serialization has been voted on and is part of the spec, sort of). > > > but only if implementations are required to do this > > While I would actually love lifting extensions in implementations having > spent quite a lot of time tracking down over- and under- propagated field > metadata, I don't think where the extension name and metadata live in the > JSON structure matters for this: deserializers that have a registry can > query it a bit more easily, and deserializers that don't can put it in > field metadata. Serializers would have to special-case those two metadata > keys. All that said, just making "metadata" an object instead of a list of > key values leads to pretty equivalent extraction of an extension parameter > (f["metadata"]["ARROW:extension:metadata"]["crs"] vs > f["type"]["metadata"]["crs"], or something). > > > But I think "simple" string types is reasonable for unparameterized > types. > > I hope we do go this direction...I dug up the testing JSON reader for the > last time I had to parse that [2] and while it's not terrible, it's also > not great if we can avoid it. > > [1] https://geoarrow.org/extension-types.html#extension-metadata > [2] > > https://github.com/apache/arrow-nanoarrow/blob/a5d7011ff4838bd4cbf70c6f72b7078ea8842cd4/src/nanoarrow/testing/testing.cc#L1044-L1099 > > On Fri, Jul 10, 2026 at 5:54 PM David Li <[email protected]> wrote: > > > To practically implement inline JSON for extension types, though, would > we > > be changing (breaking?) all the extension APIs in Arrow implementations > to > > somehow flag that they use JSON? And in any case, if some Arrow > > implementation doesn't handle this, or the extension isn't registered, > etc. > > you could still end up having a string instead of JSON and would need to > > handle it right? I don't think implementations will guarantee that JSON > > metadata always gets serialized as JSON. > > > > Hence while it seems convenient I'd like to avoid trying to special-case > > extension types and JSON. > > > > I could be convinced that extension types should be lifted to the type > and > > not the metadata, but only if implementations are required to do this (so > > no extension type metadata in the metadata, ever) again because the > > inconsistency would otherwise make things annoying (I already run into > this > > with Arrow implementations where only registered extension types get > > promoted to actual types, leading to duplicate logic/checking all over > the > > place). > > > > (FWIW: Java does actually enumerate bitwidth and signedness explicitly; > it > > also puts child fields on the field and not the type, both choices > matching > > Flatbuffers exactly. But I think "simple" string types is reasonable for > > unparameterized types.) > > > > On Sat, Jul 11, 2026, at 00:41, Dewey Dunnington wrote: > > > For at least the "uint32" abbreviation, this is how most if not all > > > implementations actually enumerate the type (not integer + signedness + > > > bitwidth), so I think that particular change makes the parsers simpler > in > > > addition to being more compact. > > > > > > For extension types that use JSON, this requires a very awkward path to > > > inspecting the type, perhaps before an Arrow implementation is > involved. > > > Maybe > > > > > > "type": {"name": "extension", "extension:name": "geoarrow.wkb", > > > "extension:metadata": {...}} > > > > > > ...would make API responses that contain extension types sufficiently > > > compact that a flat list of named key/value objects doesn't matter. > With > > > the metadata as an object instead of a list and the value field as any > > JSON > > > value I think it's still sufficiently compact to pluck an extension > > > parameter. The alternative is sufficiently gross (loop through named > key > > > and pick the first? or last? and re-parse escaped JSON?) that I am not > > sure > > > I would actually use it on the day I need to return a schema with > > GeoArrow > > > types in a JSON API response (also: this day may never come :)) > > > > > > -dewey > > > > > > On Wed, Jul 8, 2026 at 9:16 PM David Li <[email protected]> wrote: > > > > > >> I think there's value, though, in having a representation that is (1) > > >> unambiguous and consistent (2) reasonably human-friendly and > > >> machine-friendly, even if it's not the most compact. This would be > > useful > > >> for (REST) APIs and ADBC, and I don't think the compactness of the > > >> representation is as important there. What kinds of use cases are you > > >> envisioning? > > >> > > >> FWIW, we debated the JSON metadata question - I kind of wanted this > too, > > >> but it raises too many questions about what happens to the canonical > > >> representation/equality/serialization/round-tripping. (For example: we > > need > > >> to stringify the JSON to pass it to the type parser, and a round-trip > > would > > >> possibly present semantically equivalent JSON but different JSON > > strings. > > >> JSON doesn't _disallow_ duplicate keys, so what if we get different > > >> behavior? etc.) > > >> > > >> Ultimately the motivation (for me) is to use this in ADBC where we > want > > to > > >> represent Arrow schemas inside Arrow data (where serialized IPC > schemas > > >> seem to be unpalatable), and so being well-defined is most important > (I > > >> don't really want to handroll something that is only used in ADBC). > > >> > > >> -David > > >> > > >> On Wed, Jul 8, 2026, at 04:36, Dewey Dunnington wrote: > > >> > Thank you for drafting! > > >> > > > >> > I gave a few comments in the community call last Wednesday, but I'll > > >> > echo them here for discussion. None of these are blocking concerns, > > >> > just thoughts having seen this particular concept be reinvented in a > > >> > lot of places. > > >> > > > >> > While this does a great job of losslessly translating flatbuffer > > >> > schemas into JSON and back again, flatbuffer schemas are rather > > >> > awkwardly laid out and the human typability/readability is not > great. > > >> > I think there is an opportunity for this JSON spec to replace a lot > of > > >> > bespoke data type / field / schema representations, but I worry that > > >> > it's a little verbose in its current form to fill that niche. > > >> > > > >> > Just as an example, a schema with a single uint32 column with this > > >> proposal is: > > >> > > > >> > {"arrowSchema": "1.5.0", "fields": [{"name": "col0", "nullable": > true, > > >> > "type": {"name": "int", "bitWidth": 32, "isSigned": false}}]} > > >> > > > >> > ...whereas with a bit of language (apply defaults for nullability, > > >> > optional/independent versioning, special case unparameterized types) > > >> > this could be more like: > > >> > > > >> > [{"name": "col0", "type": "uint32"}] > > >> > > > >> > ...drawing from BigQuery [1] here), although that does open up a > > >> > significantly more surface area for bikeshedding on which strings > map > > >> > to which type (unless we want to reuse the C data interface ones, > > >> > which from the proposal it seems like maybe we don't). > > >> > > > >> > Another opportunity to reduce verbosity is metadata, which for an > > >> > extension type with JSON metadata would be > > >> > > > >> > [{"key": "ARROW:extension:name", "value": "geoarrow.wkb"}, {"key": > > >> > "ARROW:extension:metadata", "value": "{\"crs\": \"OGC:CRS84\"}"}] > > >> > > > >> > ...and could be more readable: > > >> > > > >> > {"ARROW:extension:name": "geoarrow.wkb", "ARROW:extension:metadata": > > >> > {"crs": "OGC:CRS84"}} > > >> > > > >> > Basically, if the goal is human readability, I think it's worth the > > >> > effort to do the bikeshedding on how to represent these concepts > that > > >> > mirror how they are printed/enumerated in implementations (not > > >> > necessarily the .fbs files). If the goal is machine readability > within > > >> > existing arrow implementations, perfectly mirroring flatbuffers JSON > > >> > or the C data interface would be less work (more opportunity for > > >> > reusing existing functions). > > >> > > > >> > Cheers, > > >> > > > >> > -dewey > > >> > > > >> > [1] > > >> > > > >> > > > https://docs.cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Schema#Google__Cloud__Bigquery__Schema_load_class_ > > >> > > > >> > On Thu, Jul 2, 2026 at 3:37 PM Kent Wu <[email protected]> wrote: > > >> >> > > >> >> Hi all, > > >> >> > > >> >> I'd like to raise a topic for discussion that has surfaced a few > > >> >> separate times over the years in the Arrow community, which is the > > >> >> lack of a canonical human-readable representation of Arrow schemas. > > >> >> > > >> >> Arrow schemas today only canonically serialize as IPC binary, which > > is > > >> >> a friction point for application-level tasks where binary blobs are > > >> >> not ergonomic, such as JSON API contracts, hand authoring or > reading, > > >> >> and persistence use cases, to name a few. > > >> >> > > >> >> The motivating pain point right now is at the ADBC metadata > boundary. > > >> >> The 1.2 milestone includes proposals for new metadata APIs, and > > >> >> several contributors have noted that returning Arrow schemas via > IPC > > >> >> is unsatisfactory for this use case. > > >> >> > > >> >> ADBC 1.2 Milestone > > >> >> - https://github.com/apache/arrow-adbc/milestone/9 > > >> >> > > >> >> Discussion within various issues: > > >> >> - https://github.com/apache/arrow-adbc/issues/4400 > > >> >> - https://github.com/apache/arrow-adbc/issues/1514 > > >> >> - https://github.com/apache/arrow-adbc/issues/1704 > > >> >> - https://github.com/apache/arrow-adbc/pull/4031 > > >> >> > > >> >> As a starting point, I've put together a proposal for how a JSON > > >> >> representation might be structured. > > >> >> > > >> >> Rather than dump the whole thing here, I've put it in this google > doc > > >> >> which is open for comments: > > >> >> > > >> >> > > >> > > > https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing > > >> >> > > >> >> It's early and open to change, so I'd welcome feedback of any kind. > > >> >> > > >> >> Looking forward to hearing your thoughts > > >> > > >
