> we either have to complicate how we define equality or the JSON representation can't round > trip without significant special casing for implementations with extension > types which we're trying to avoid
The equality of JSON objects within a metadata value JSON string is never guaranteed, nor is roundtripping (e.g., Arrow C++ would instantiate an ExtensionType instance and reserialize it, potentially putting keys in a different order). Enabling metadata values to be any valid JSON would improve both equality and roundtripping (because the JSON object, not its stringified version, could be compared for equality). The extra complexity is on readers, who when they see a metadata value that is not a string have to serialize it as one (this is what we do for GeoArrow -> Parquet and I believe it's a two-liner). Writers aren't required to carry extra complexity, they just aren't prevented from writing idiomatic JSON if they would like to (they could be warned that such values may be serialized on roundtrip if that is important to a specific producer). Many writers already special case extension types on both import and export to generate/parse field metadata...this is not a new code path. > I wouldn't object to making metadata an object. If the goal is idiomatic JSON, I think this would be a good choice. Cheers, -dewey On Sat, Jul 11, 2026 at 12:41 PM Matt Topol <[email protected]> wrote: > > The language here could be similar...implementations are not required to > serialize them any differently but are allowed to produce nicer looking > JSON for extension types when they can guarantee it's valid (many > implementations can do this, at least for canonical extensions whose > serialization has been voted on and is part of the spec, sort of). > > The problem i have with this is just that it means we either have to > complicate how we define equality or the JSON representation can't round > trip without significant special casing for implementations with extension > types which we're trying to avoid. I just don't think the convenience of > nicer looking JSON is worth the complexity. i.e. I agree with what David > said earlier in this thread. > > > All that said, just making "metadata" an object instead of a list of key > values leads to pretty equivalent extraction of an extension parameter > > We discussed this a bit and landed on using the list of key values to be > more like the IPC representation, but I wouldn't object to making metadata > an object. > > --Matt > > On Fri, Jul 10, 2026, 10:47 PM Dewey Dunnington < > [email protected]> > wrote: > > > > To practically implement inline JSON for extension types, though > > > > The way we do this in GeoArrow (Arrow Go, Arrow C++ Parquet, > > arrow-rs/parquet) to prevent escaped-json-in-json when going from Parquet > > (crs is a string) -> GeoArrow (crs is a member of a JSON object) is to > > parse the value, and if it parses as a JSON object, use that instead of > the > > stringified version. The language we have in the spec for this is > > "Producers should not write an escaped JSON object to the crs key" [1]. > The > > language here could be similar...implementations are not required to > > serialize them any differently but are allowed to produce nicer looking > > JSON for extension types when they can guarantee it's valid (many > > implementations can do this, at least for canonical extensions whose > > serialization has been voted on and is part of the spec, sort of). > > > > > but only if implementations are required to do this > > > > While I would actually love lifting extensions in implementations having > > spent quite a lot of time tracking down over- and under- propagated field > > metadata, I don't think where the extension name and metadata live in the > > JSON structure matters for this: deserializers that have a registry can > > query it a bit more easily, and deserializers that don't can put it in > > field metadata. Serializers would have to special-case those two metadata > > keys. All that said, just making "metadata" an object instead of a list > of > > key values leads to pretty equivalent extraction of an extension > parameter > > (f["metadata"]["ARROW:extension:metadata"]["crs"] vs > > f["type"]["metadata"]["crs"], or something). > > > > > But I think "simple" string types is reasonable for unparameterized > > types. > > > > I hope we do go this direction...I dug up the testing JSON reader for the > > last time I had to parse that [2] and while it's not terrible, it's also > > not great if we can avoid it. > > > > [1] https://geoarrow.org/extension-types.html#extension-metadata > > [2] > > > > > https://github.com/apache/arrow-nanoarrow/blob/a5d7011ff4838bd4cbf70c6f72b7078ea8842cd4/src/nanoarrow/testing/testing.cc#L1044-L1099 > > > > On Fri, Jul 10, 2026 at 5:54 PM David Li <[email protected]> wrote: > > > > > To practically implement inline JSON for extension types, though, would > > we > > > be changing (breaking?) all the extension APIs in Arrow implementations > > to > > > somehow flag that they use JSON? And in any case, if some Arrow > > > implementation doesn't handle this, or the extension isn't registered, > > etc. > > > you could still end up having a string instead of JSON and would need > to > > > handle it right? I don't think implementations will guarantee that JSON > > > metadata always gets serialized as JSON. > > > > > > Hence while it seems convenient I'd like to avoid trying to > special-case > > > extension types and JSON. > > > > > > I could be convinced that extension types should be lifted to the type > > and > > > not the metadata, but only if implementations are required to do this > (so > > > no extension type metadata in the metadata, ever) again because the > > > inconsistency would otherwise make things annoying (I already run into > > this > > > with Arrow implementations where only registered extension types get > > > promoted to actual types, leading to duplicate logic/checking all over > > the > > > place). > > > > > > (FWIW: Java does actually enumerate bitwidth and signedness explicitly; > > it > > > also puts child fields on the field and not the type, both choices > > matching > > > Flatbuffers exactly. But I think "simple" string types is reasonable > for > > > unparameterized types.) > > > > > > On Sat, Jul 11, 2026, at 00:41, Dewey Dunnington wrote: > > > > For at least the "uint32" abbreviation, this is how most if not all > > > > implementations actually enumerate the type (not integer + > signedness + > > > > bitwidth), so I think that particular change makes the parsers > simpler > > in > > > > addition to being more compact. > > > > > > > > For extension types that use JSON, this requires a very awkward path > to > > > > inspecting the type, perhaps before an Arrow implementation is > > involved. > > > > Maybe > > > > > > > > "type": {"name": "extension", "extension:name": "geoarrow.wkb", > > > > "extension:metadata": {...}} > > > > > > > > ...would make API responses that contain extension types sufficiently > > > > compact that a flat list of named key/value objects doesn't matter. > > With > > > > the metadata as an object instead of a list and the value field as > any > > > JSON > > > > value I think it's still sufficiently compact to pluck an extension > > > > parameter. The alternative is sufficiently gross (loop through named > > key > > > > and pick the first? or last? and re-parse escaped JSON?) that I am > not > > > sure > > > > I would actually use it on the day I need to return a schema with > > > GeoArrow > > > > types in a JSON API response (also: this day may never come :)) > > > > > > > > -dewey > > > > > > > > On Wed, Jul 8, 2026 at 9:16 PM David Li <[email protected]> wrote: > > > > > > > >> I think there's value, though, in having a representation that is > (1) > > > >> unambiguous and consistent (2) reasonably human-friendly and > > > >> machine-friendly, even if it's not the most compact. This would be > > > useful > > > >> for (REST) APIs and ADBC, and I don't think the compactness of the > > > >> representation is as important there. What kinds of use cases are > you > > > >> envisioning? > > > >> > > > >> FWIW, we debated the JSON metadata question - I kind of wanted this > > too, > > > >> but it raises too many questions about what happens to the canonical > > > >> representation/equality/serialization/round-tripping. (For example: > we > > > need > > > >> to stringify the JSON to pass it to the type parser, and a > round-trip > > > would > > > >> possibly present semantically equivalent JSON but different JSON > > > strings. > > > >> JSON doesn't _disallow_ duplicate keys, so what if we get different > > > >> behavior? etc.) > > > >> > > > >> Ultimately the motivation (for me) is to use this in ADBC where we > > want > > > to > > > >> represent Arrow schemas inside Arrow data (where serialized IPC > > schemas > > > >> seem to be unpalatable), and so being well-defined is most important > > (I > > > >> don't really want to handroll something that is only used in ADBC). > > > >> > > > >> -David > > > >> > > > >> On Wed, Jul 8, 2026, at 04:36, Dewey Dunnington wrote: > > > >> > Thank you for drafting! > > > >> > > > > >> > I gave a few comments in the community call last Wednesday, but > I'll > > > >> > echo them here for discussion. None of these are blocking > concerns, > > > >> > just thoughts having seen this particular concept be reinvented > in a > > > >> > lot of places. > > > >> > > > > >> > While this does a great job of losslessly translating flatbuffer > > > >> > schemas into JSON and back again, flatbuffer schemas are rather > > > >> > awkwardly laid out and the human typability/readability is not > > great. > > > >> > I think there is an opportunity for this JSON spec to replace a > lot > > of > > > >> > bespoke data type / field / schema representations, but I worry > that > > > >> > it's a little verbose in its current form to fill that niche. > > > >> > > > > >> > Just as an example, a schema with a single uint32 column with this > > > >> proposal is: > > > >> > > > > >> > {"arrowSchema": "1.5.0", "fields": [{"name": "col0", "nullable": > > true, > > > >> > "type": {"name": "int", "bitWidth": 32, "isSigned": false}}]} > > > >> > > > > >> > ...whereas with a bit of language (apply defaults for nullability, > > > >> > optional/independent versioning, special case unparameterized > types) > > > >> > this could be more like: > > > >> > > > > >> > [{"name": "col0", "type": "uint32"}] > > > >> > > > > >> > ...drawing from BigQuery [1] here), although that does open up a > > > >> > significantly more surface area for bikeshedding on which strings > > map > > > >> > to which type (unless we want to reuse the C data interface ones, > > > >> > which from the proposal it seems like maybe we don't). > > > >> > > > > >> > Another opportunity to reduce verbosity is metadata, which for an > > > >> > extension type with JSON metadata would be > > > >> > > > > >> > [{"key": "ARROW:extension:name", "value": "geoarrow.wkb"}, {"key": > > > >> > "ARROW:extension:metadata", "value": "{\"crs\": \"OGC:CRS84\"}"}] > > > >> > > > > >> > ...and could be more readable: > > > >> > > > > >> > {"ARROW:extension:name": "geoarrow.wkb", > "ARROW:extension:metadata": > > > >> > {"crs": "OGC:CRS84"}} > > > >> > > > > >> > Basically, if the goal is human readability, I think it's worth > the > > > >> > effort to do the bikeshedding on how to represent these concepts > > that > > > >> > mirror how they are printed/enumerated in implementations (not > > > >> > necessarily the .fbs files). If the goal is machine readability > > within > > > >> > existing arrow implementations, perfectly mirroring flatbuffers > JSON > > > >> > or the C data interface would be less work (more opportunity for > > > >> > reusing existing functions). > > > >> > > > > >> > Cheers, > > > >> > > > > >> > -dewey > > > >> > > > > >> > [1] > > > >> > > > > >> > > > > > > https://docs.cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Schema#Google__Cloud__Bigquery__Schema_load_class_ > > > >> > > > > >> > On Thu, Jul 2, 2026 at 3:37 PM Kent Wu <[email protected]> > wrote: > > > >> >> > > > >> >> Hi all, > > > >> >> > > > >> >> I'd like to raise a topic for discussion that has surfaced a few > > > >> >> separate times over the years in the Arrow community, which is > the > > > >> >> lack of a canonical human-readable representation of Arrow > schemas. > > > >> >> > > > >> >> Arrow schemas today only canonically serialize as IPC binary, > which > > > is > > > >> >> a friction point for application-level tasks where binary blobs > are > > > >> >> not ergonomic, such as JSON API contracts, hand authoring or > > reading, > > > >> >> and persistence use cases, to name a few. > > > >> >> > > > >> >> The motivating pain point right now is at the ADBC metadata > > boundary. > > > >> >> The 1.2 milestone includes proposals for new metadata APIs, and > > > >> >> several contributors have noted that returning Arrow schemas via > > IPC > > > >> >> is unsatisfactory for this use case. > > > >> >> > > > >> >> ADBC 1.2 Milestone > > > >> >> - https://github.com/apache/arrow-adbc/milestone/9 > > > >> >> > > > >> >> Discussion within various issues: > > > >> >> - https://github.com/apache/arrow-adbc/issues/4400 > > > >> >> - https://github.com/apache/arrow-adbc/issues/1514 > > > >> >> - https://github.com/apache/arrow-adbc/issues/1704 > > > >> >> - https://github.com/apache/arrow-adbc/pull/4031 > > > >> >> > > > >> >> As a starting point, I've put together a proposal for how a JSON > > > >> >> representation might be structured. > > > >> >> > > > >> >> Rather than dump the whole thing here, I've put it in this google > > doc > > > >> >> which is open for comments: > > > >> >> > > > >> >> > > > >> > > > > > > https://docs.google.com/document/d/1ho0FKy9ge0tUSRzebq1AFi28KR1H_utecQjHx4CNCEs/edit?usp=sharing > > > >> >> > > > >> >> It's early and open to change, so I'd welcome feedback of any > kind. > > > >> >> > > > >> >> Looking forward to hearing your thoughts > > > >> > > > > > >
