Hi Yufei, I'm fine with JSON response payloads following the Apache Ossie JSON structure in v1 API and deferring other payload types to future API revisions.
I'm not up-to-date with Ossie though... Is it explicit about the spec version in its JSON data representation? If not, then Polaris will have to indicate the version of the Ossie spec somehow. This will probably require an envelope in the response anyway. The alternative is to revision the Polaris API (v1, v2, v3, etc.) for every Ossie spec change, which seems impractical to me. WDYT? Thanks, Dmitri. On Tue, Jul 21, 2026 at 9:37 PM Yufei Gu <[email protected]> wrote: > Thanks for the discussion. > > Dmitri, could you clarify what the envelope would look like? > > My current understanding is that it might look roughly like this: > > { > "format": "ossie", > "version": "1.0", > "payload": { > "...": "..." > } > } > > For a non JSON format, the payload would instead need to be > represented as an encoded string: > > { > "format": "okf", > "version": "1.0", > "payload": "encoded document content" > } > > If that is the intended design, I am not sure the envelope gives us > much beyond option 1, as clients still need format specific handling. > > So the clearest choices seem to be either: > 1. Direct JSON for Ossie, with no envelope, that seems to be what JB > proposed. In that case, we'd have to change the spec or introduce a v2 > endpoint when we introduce a new format like OKF. > 2. a format-agnostic envelope with an opaque string payload. > > Am I missing another benefit of the envelope approach? > > Thanks, > Yufei > > Yufei > > > On Thu, Jul 16, 2026 at 4:22 AM Jean-Baptiste Onofré <[email protected]> > wrote: > > > > Hi everyone, > > > > I prefer to store and expose the semantic API using a direct JSON > > payload format ("raw"). > > > > I also suggest that we focus exclusively on the Apache Ossie format > > for now, without any wrapping. > > > > Regarding the evolution of the Ossie specification, I believe it is > > better to store raw JSON and handle the plumbing on our end. This > > approach allows any tool, including AI tools, to directly consume the > > Ossie definition. > > > > Regards, > > JB > > > > On Mon, Jul 13, 2026 at 8:13 PM Yufei Gu <[email protected]> wrote: > > > > > > Hi Dmitri, Thanks! I think option 2 is a reasonable choice. The > advantage I > > > see with option 1 is future extensibility. If we later decide to > support > > > another semantic model format that isn't JSON, such as OKF, option 2 > would > > > require a REST spec change, while option 1 would not. That's why I > slightly > > > prefer option 1, although I think either option is much better than > > > modeling the semantic model structure directly in the REST API. What > do you > > > think? > > > > > > Thanks, > > > Yufei > > > > > > > > > On Mon, Jul 13, 2026 at 11:10 AM Yufei Gu <[email protected]> > wrote: > > > > > > > Hi Robert, > > > > > > > > Thanks for the thoughtful feedback. > > > > > > > > I agree that the current API is narrower than a complete semantic > layer. > > > > Today it's primarily about managing the lifecycle of semantic model > > > > documents. Discovery, search, and client consumption are all > important > > > > topics, but I think they're broader than the Ossie semantic model > itself. > > > > We touched on some of these in the last community sync[1], and I'm > happy to > > > > continue that discussion in a separate thread. > > > > > > > > I also agree that we should be careful not to overstate what this > initial > > > > API provides. Calling it a beta semantic model registry or document > hosting > > > > API seems reasonable. > > > > > > > > That said, I think those questions are orthogonal to the payload > > > > representation discussion. My original topic was simply whether the > REST > > > > API should treat the semantic model as a raw document, an opaque JSON > > > > payload, or model the schema directly. Since the consumption story > is still > > > > evolving, I think keeping the REST contract loosely coupled to the > > > > underlying semantic model specification gives us the most > flexibility. > > > > > > > > [1] > > > > > https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing > > > > > > > > Thanks, > > > > > > > > Yufei > > > > > > > > On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <[email protected]> wrote: > > > > > > > >> Hi, > > > >> > > > >> I think the payload representation question depends on the client > model we > > > >> expect this API to support. > > > >> > > > >> I support Polaris hosting Apache Ossie semantic-model documents as > a beta > > > >> foundation. That seems like a useful first step while Ossie itself > is > > > >> still > > > >> evolving. > > > >> > > > >> But I do not think we should describe the current API as enabling AI > > > >> tools, > > > >> BI tools, or human semantic-model discovery yet. > > > >> > > > >> The merged API is primarily namespace/name CRUD for a semantic-model > > > >> document. > > > >> That is opaque document storage and exact point-retrieval. > > > >> > > > >> It does not enable clients to discover the right semantic model > from a > > > >> query, > > > >> nor to find semantic models by table, metric, domain, user, or > capability. > > > >> > > > >> It also does not define a standard consuming API or tool contract, > > > >> search/indexing contract, freshness model, or > current/trusted/certified > > > >> model > > > >> semantics. > > > >> > > > >> That distinction matters because the REST API is the user-facing > contract. > > > >> > > > >> If the beta API is intended only as opaque document hosting, I > think the > > > >> spec > > > >> and docs should say that clearly, and users/clients should not infer > > > >> broader > > > >> discovery or interoperability semantics from the CRUD API. > > > >> > > > >> I am not asking to solve the full semantic-layer story immediately. > > > >> > > > >> I am asking that durable implementation work does not get ahead of > the > > > >> client-consumption story. > > > >> > > > >> The client model should drive the persistent data model, not the > other way > > > >> around. > > > >> > > > >> Once semantic models are stored as durable Polaris entities, choices > > > >> around > > > >> identity, versioning, validation, indexing, size limits, > source-table > > > >> references, and freshness become much harder to change. > > > >> > > > >> For now, I would be comfortable describing this as beta Apache Ossie > > > >> document > > > >> hosting / semantic-model registry work. > > > >> > > > >> I would not yet be comfortable describing it as enabling AI, BI, or > human > > > >> semantic workflows until the discovery and client-consumption story > > > >> exists. > > > >> > > > >> Robert > > > >> > > > >> > > > >> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov < > [email protected]> > > > >> wrote: > > > >> > > > >> > Hi Yufei, > > > >> > > > > >> > Thanks for starting this thread! > > > >> > > > > >> > I tend to think it is best to represent Ossie data as direct JSON > > > >> without > > > >> > defining its structure in the Polaris OpenAPI spec (Ossie schemas > are > > > >> > controlled by Ossie, not Polaris). I believe this corresponds to > Option > > > >> 2 > > > >> > from your email. > > > >> > > > > >> > Polaris code that implements the new API will then interpret the > Ossie > > > >> > parts according to the declared version of the Ossie spec. > > > >> > > > > >> > With that in mind, the API should clearly state the format (Ossie > or > > > >> OKF) > > > >> > and the specification version of the semantic data sub-object > > > >> (apologies if > > > >> > it has that already, I'm behind on the related PR updates). > Obviously > > > >> OKF > > > >> > will have a different representation in the payload, but this > should not > > > >> > prevent Ossie from leveraging JSON synergies. > > > >> > > > > >> > Anand's work on solving a similar problem in the Metrics API > [4115] may > > > >> be > > > >> > reusable here. > > > >> > > > > >> > [4115] https://github.com/apache/polaris/pull/4115 > > > >> > > > > >> > Cheers, > > > >> > Dmitri. > > > >> > > > > >> > > > > >> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <[email protected]> > wrote: > > > >> > > > > >> > > Hi folks, > > > >> > > > > > >> > > Following JB's suggestion, I'd like to start a dedicated > discussion on > > > >> > the > > > >> > > REST API payload representation for semantic models. > > > >> > > > > > >> > > I think there are three possible approaches: > > > >> > > > > > >> > > 1. > > > >> > > > > > >> > > Represent the semantic model as a raw string. > > > >> > > 2. > > > >> > > > > > >> > > Represent the semantic model as an opaque JSON document. > > > >> > > 3. > > > >> > > > > > >> > > Model the semantic model structure directly in the REST > > > >> specification. > > > >> > > > > > >> > > I think it's helpful to separate the REST API from Polaris' > internal > > > >> > > representation. The REST API is the long-term contract with > clients, > > > >> > while > > > >> > > the internal representation can evolve independently. > > > >> > > > > > >> > > I'm comfortable with either option 1 or option 2. Both avoid > coupling > > > >> the > > > >> > > REST API to the Ossie schema and allow Polaris to validate the > payload > > > >> > > based on the semantic model type and version while preserving > the > > > >> > document > > > >> > > through write and read operations. > > > >> > > > > > >> > > My concern is with option 3. Since the Ossie schema is > versioned and > > > >> > > expected to evolve, modeling the full semantic model structure > > > >> directly > > > >> > in > > > >> > > the REST specification would tightly couple the Polaris REST > API to > > > >> Ossie > > > >> > > versions. Every Ossie schema evolution could require changes to > the > > > >> REST > > > >> > > specification, generated clients, and potentially client > applications. > > > >> > > > > > >> > > Between options 1 and 2, I think there is an additional > tradeoff. > > > >> > > > > > >> > > An opaque JSON document assumes that semantic models are always > > > >> > represented > > > >> > > as JSON. While that works well for Ossie today, Polaris may > support > > > >> other > > > >> > > semantic model formats in the future. For example, OKF[1] is > defined > > > >> as > > > >> > > Markdown rather than JSON. Using a raw string keeps the REST API > > > >> > > independent of any particular document format, allowing Polaris > to > > > >> > support > > > >> > > JSON, Markdown, or other representations without changing the > API > > > >> > contract. > > > >> > > > > > >> > > So my current view is: > > > >> > > > > > >> > > - > > > >> > > > > > >> > > Option 1 provides the greatest flexibility and is format > agnostic. > > > >> > > - > > > >> > > > > > >> > > Option 2 is a natural choice if we want to optimize > specifically > > > >> for > > > >> > > JSON based semantic models. > > > >> > > - > > > >> > > > > > >> > > Option 3 provides strong typing, but at the cost of coupling > the > > > >> REST > > > >> > > API to Ossie schema evolution. > > > >> > > > > > >> > > I'm happy with either option 1 or option 2, but I'd avoid > option 3 for > > > >> > the > > > >> > > reasons above. > > > >> > > > > > >> > > Thoughts? > > > >> > > > > > >> > > 1. > > > >> > > > > > >> > > > > > >> > > > > >> > https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing > > > >> > > > > > >> > > Thanks, > > > >> > > > > > >> > > Yufei > > > >> > > > > > >> > > > > >> > > > > >
