Hi Dmitri, Thanks - Noted. +1 on lowercasing type..
I'll open the prerequisite PR next (mapper next to the SPI in runtime/service). Thanks *Vignesh* On Tue, 6 Oct 2026 at 21:32, Dmitri Bourlatchkov <[email protected]> wrote: > Hi Vignesh, > > The proposal LGTM. > > Minor comment: SInce CloudEvents apparently prefer lowercase names, shall > we lowercase the type namne (in JSON)? > > Re: backward compatibility for Kafka. Having a flag to send old payloads > works for me. However, I believe this tends to prolong adoption and users > have to face a hard change anyway (when their admin flips the switch). This > may not be suitable for rolling upgrade scenarios. That said, we can > probably debate this later, after establishing the common format. > > Cheers, > Dmitri. > > On Tue, Oct 6, 2026 at 8:11 AM vignesh a <[email protected]> wrote: > > > Hi Dmitri, > > > > Thanks - good questions, and yes at a high level that's the direction. > > Wanted to answer them against what JB and I already talked through, and > > I've added the draft v1 JSON again so we're looking at the same thing. > > > > 1) Same JSON for all JSON-producing listeners? > > > > For the transport listeners in this thread (Kafka, webhook, CloudWatch), > > yes - same CloudEvents-shaped payload is the end goal. OpenTelemetry and > > the persistence listener keep their own representations for now; those > > stay out of scope for the first PR. > > > > 2) Adhere to CloudEvents? > > > > Yes. v1 means CloudEvents 1.0 context attributes at the top only, and the > > Polaris fields in "data". That also keeps us honest with CE rules on > > context attrs (lowercase alphanumerics / scalars) - so things like > > activated_roles live in "data", not as top-level CE extensions. If a > > newer CE version becomes relevant later we can bump then.. > > > > 3) Migrating existing consumers that aren't 1:1 with the new shape? > > > > Union / dual-publish for a few releases is one way to do it. On the list > > JB leaned toward a per-listener format option instead (today's flat Kafka > > payload as the default, shared envelope opt-in), mainly so we don't > > double volume on the topic. That felt right to me too for 1.7 / 1.8 > > consumers - they keep working until they flip the knob. Webhook [1] can > > land on the shared envelope from day one since it isn't released that > > way yet. Open to other ideas if you see a cleaner path.. LMK > > > > 4) Dedicated SPI module? > > > > Also +1 (Alex mentioned the same). Kafka depending on runtime/service > > just for the SPI is tricky, and having the mapper live in that SPI > > module later makes a lot of sense. For the first PR I was thinking of > > putting the shared mapper next to the existing events SPI in > > runtime/service, then extracting SPI + mapper as a follow-up - mostly to > > avoid getting stuck on module cycles before the envelope lands (JB had > > flagged that). Will do the SPI split earlier if that feels better. WDYT > ?? > > > > Draft v1 JSON (CE context + today's Kafka body in "data"): > > > > { > > "specversion": "1.0", > > "id": "550e8400-e29b-41d4-a716-446655440000", > > "type": "org.apache.polaris.AFTER_CREATE_TABLE", > > "source": "org.apache.polaris", > > "time": "2026-10-04T12:34:56.789Z", > > "datacontenttype": "application/json", > > "data": { > > "event_type": "AFTER_CREATE_TABLE", > > "event_id": "550e8400-e29b-41d4-a716-446655440000", > > "timestamp": "2026-10-04T12:34:56.789Z", > > "realm_id": "default-realm", > > "principal": "alice", > > "activated_roles": ["catalog_admin"], > > "request_id": "req-123", > > "catalog_name": "sales", > > "namespace": ["analytics"], > > "table_name": "orders", > > "table_identifier": ["analytics", "orders"] > > } > > } > > > > A couple notes: > > - "data" is basically today's Kafka EventToJsonMapper body, so id/time > > show up both as CE context and again inside data (event_id / > > timestamp) for Kafka parity. > > - Type is a namespaced PolarisEventType name; source is a constant for > > now - happy to bike-shed those if needed. > > - Webhook delivery time stays transport metadata (HTTP header), not in > > the shared envelope. > > - Rename events would also carry source / destination arrays in "data", > > same as Kafka today; omitted above because this example is > > AFTER_CREATE_TABLE. > > > > If this looks right, I'll open the prerequisite PR on that basis... > > > > Thanks > > *Vignesh* > > > > [1] https://github.com/apache/polaris/pull/5084 > > > > On Tue, 6 Oct 2026 at 05:53, Dmitri Bourlatchkov <[email protected]> > wrote: > > > > > Hi Vgnesh, > > > > > > Your proposal sounds good to be at high level. > > > > > > Just to clarify: is our end goal here to use the same JSON payload for > > all > > > event listeners that produce JSON? > > > > > > Are we going to adhere to the CloudEvents spec for this JSON? Obviously > > > starting with 1.0 and migrating to newer version as the become > > available... > > > Is that the plan? > > > > > > How are we going to migrare existing use cases that are not 1:1 with > > > the ideal JSON schema? Will be publish a union of attributes for a few > > > releases to allow consumers to switch to the new schema? > > > > > > +1 to adding a dedicated SPI module. I suppose it can also contain the > > > common java-to-JSON mapping code. > > > > > > Thanks, > > > Dmitri. > > > > > > On Mon, Oct 5, 2026 at 4:29 AM Alex Dutra <[email protected]> wrote: > > > > > > > Hi all, > > > > > > > > +1 for the proposed v1 JSON, and +1 for having Kafka parity in v1. > > > > > > > > I also welcome the idea of isolating the Events SPI in its own > module. > > > > Having the Kafka module depend on runtime/service just for the SPI is > > > > concerning. > > > > > > > > Looking forward to the first PR! > > > > > > > > Thanks, > > > > Alex > > > > > > > > On Mon, Oct 5, 2026 at 10:02 AM vignesh a <[email protected]> > > > wrote: > > > > > > > > > > Hi JB, > > > > > > > > > > Thanks - this is really useful, and I'm with you on the sequencing > > > > > (shared envelope first, then rebase the webhook). > > > > > > > > > > Your points on shape make sense to me too: > > > > > 1. CloudEvents context attrs at the top only, Polaris payload in > > "data" > > > > > (so things like activated_roles stay valid) > > > > > 2. reuse the existing EventAttributes / Kafka snake_case keys in > > "data" > > > > > 3. v1 at Kafka parity, not the thinner webhook set I suggested > > earlier > > > > > 4. mapper next to the events SPI in runtime/service for now > > > > > 5. deliverytime as webhook HTTP metadata, not in the shared > envelope > > > > > 6. OpenTelemetry + persistence listeners out of scope for this > > > > > > > > > > Also +1 on settling Kafka compat on the list. per-listener format > > > option > > > > > with today's flat payload as the default (shared envelope opt-in) > > > sounds > > > > > better than dual-publish. > > > > > > > > > > Draft v1 JSON I have in mind: > > > > > > > > > > { > > > > > "specversion": "1.0", > > > > > "id": "550e8400-e29b-41d4-a716-446655440000", > > > > > "type": "org.apache.polaris.AFTER_CREATE_TABLE", > > > > > "source": "org.apache.polaris", > > > > > "time": "2026-10-04T12:34:56.789Z", > > > > > "datacontenttype": "application/json", > > > > > "data": { > > > > > "event_type": "AFTER_CREATE_TABLE", > > > > > "event_id": "550e8400-e29b-41d4-a716-446655440000", > > > > > "timestamp": "2026-10-04T12:34:56.789Z", > > > > > "realm_id": "default-realm", > > > > > "principal": "alice", > > > > > "activated_roles": ["catalog_admin"], > > > > > "request_id": "req-123", > > > > > "catalog_name": "sales", > > > > > "namespace": ["analytics"], > > > > > "table_name": "orders", > > > > > "table_identifier": ["analytics", "orders"] > > > > > } > > > > > } > > > > > > > > > > A couple notes on purpose: > > > > > - "data" is basically today's Kafka body, so id/time show up both > as > > CE > > > > > context and again inside data (event_id / timestamp) for Kafka > > parity > > > > > - type is a namespaced PolarisEventType name; source is a constant > > for > > > > > now - will change if needed.. > > > > > - webhook would put delivery time on an HTTP header > > > > > (e.g. X-Polaris-Delivery-Time), not in the envelope > > > > > > > > > > if this is roughly the right shape, I'll draft the prerequisite PR > > > around > > > > > it. if you'd rather nail type/source naming first, tell me which > > forms > > > > > you prefer. > > > > > > > > > > Thanks, > > > > > *Vignesh* > > > > > > > > > > [1] https://github.com/apache/polaris/pull/5084 > > > > > > > > > > On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <[email protected] > > > > > > wrote: > > > > > > > > > > > Hi Vignesh > > > > > > > > > > > > Thanks for starting this thread. > > > > > > > > > > > > I'm +1 on the sequence: shared envelope first and then rebase the > > > > webhook > > > > > > PR. I slightly prefer your proposal over B or C. > > > > > > > > > > > > Some general comments: > > > > > > 1. CloudEvents requires attribute names to be lowercase > > alphanumerics > > > > and > > > > > > attribute values to be scalars afair. So a flat set of extensions > > > > doesn't > > > > > > really work for us: activatedroles as an array is not valid for > > > > > > instance. Or we have to transform somehow. I would keep only the > > > > > > CloudEvents context attributes at the top level and put the > Polaris > > > > fields > > > > > > in "data". > > > > > > 2. The EventAttributes keys already have underscored names > > > > (catalog_name, > > > > > > table_identifier, ...) and they match what Kafka emits. I would > > reuse > > > > them > > > > > > as is. Then the current Kafka body basically becomes "data", > which > > > also > > > > > > makes the migration easier. > > > > > > 3. I would not start from the thin set: migrating Kafka to it > would > > > > drop > > > > > > fields that consumers already get (namespace, catalog, view, ..). > > > Kafka > > > > > > only has nine optional fields, so I think v1 should start at > Kafka > > > > parity. > > > > > > 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I > > > think > > > > we > > > > > > should agree on the mechanism here on the list rather than in the > > PR > > > > > > description (if it doesn't happen on the mailing list, it never > > > happens > > > > > > :)). I would prefer a per-listener format option, with the > current > > > > format > > > > > > as default, over dual-publish (which doubles the volume on the > > > topic). > > > > > > 5. The events SPI and the CloudWatch listener are both in > > > > runtime/service, > > > > > > and the Kafka extension depends on it. A new module would > > introduce a > > > > cycle > > > > > > unless we extract the SPI first, so I would put the mapper next > to > > > the > > > > SPI > > > > > > in runtime/service for now, and move it when the SPI is moved. > > > > > > 6. deliverytime is a transport metadata, I would move it to a > HTTP > > > > header > > > > > > instead of the shared envelope. We should also decide what "type" > > and > > > > > > "source" look like (today it's a bare enum name and a constant). > > > > > > > > > > > > Could you share the v1 JSON you have in mind on this thread > before > > > > opening > > > > > > the PR? I think it will be easier to converge on an example. > > > > > > > > > > > > The OpenTelemetry and persistence listeners have their own > > > > representations > > > > > > as well. I'm fine keeping them out of scope, but it's worth > stating > > > it > > > > > > explicitly. > > > > > > > > > > > > Regards > > > > > > JB > > > > > > > > > > > > On Sun, Oct 4, 2026 at 11:47 AM vignesh a < > [email protected]> > > > > wrote: > > > > > > > > > > > > > \Hi all, > > > > > > > > > > > > > > Robert and EJ have both asked that we settle a shared event > > > envelope > > > > > > > before merging the webhook listener [1]. I agree with that > > > sequencing > > > > > > > and would like a quick list consensus on shape and placement > > > before I > > > > > > > open the prerequisite PR. > > > > > > > > > > > > > > What each path emits today (from current main / [1]): > > > > > > > > > > > > > > - Kafka (extensions/events/kafka/.../EventToJsonMapper): > > > > > > > flat snake_case: event_type, event_id, timestamp, realm_id, > > > > principal, > > > > > > > activated_roles, request_id, plus a fairly rich attribute set > > > > > > > (table_identifier as a string list, namespace/catalog/view > > > fields, > > > > > > > rename source/destination, and similar). > > > > > > > > > > > > > > - Webhook ([1], WebhookEventListener.toJson): > > > > > > > CloudEvents-shaped: specversion, id, type, source, time, plus > > > > > > > extensions deliverytime, realmid, tableidentifier (single > > > string), > > > > > > > principal, activatedroles, requestid. Thinner attribute > > coverage > > > > than > > > > > > > Kafka. > > > > > > > > > > > > > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener): > > > > > > > flat map: event_type, optional table_identifier (string), > > > realm_id, > > > > > > > principal, activated_roles, request_id. No event_id in the > JSON > > > > body; > > > > > > > timestamp is on InputLogEvent. > > > > > > > > > > > > > > So this is not a mechanical "extract EventToJsonMapper." > Naming, > > > > > > > structure, identifier encoding, and attribute breadth already > > > > diverge, > > > > > > > and the shared code should not live inside the kafka extension. > > > > > > > > > > > > > > Proposal: > > > > > > > > > > > > > > 1) Land a small prerequisite that introduces a shared, > versioned, > > > > > > > CloudEvents-shaped mapper (core fields + a documented > > extension > > > > set), > > > > > > > placed outside any one transport extension. > > > > > > > > > > > > > > 2) Rebase [1] onto that shared mapper. > > > > > > > > > > > > > > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit > > > > > > > compatibility plan so existing consumers are not silently > > broken > > > > > > > (version field and/or dual-publish window; exact mechanism > in > > > the > > > > > > > prerequisite PR description). > > > > > > > > > > > > > > 4) Attribute breadth for v1: start from the > > webhook/CloudWatch-thin > > > > > > > common set; add Kafka's richer attributes behind the same > > > > versioned > > > > > > > envelope rather than blocking v1 on full Kafka parity. > > > > > > > > > > > > > > If that direction is wrong, the useful alternatives seem to be: > > > > > > > > > > > > > > - B) Prefer Kafka's flat snake_case as the shared shape instead > > of > > > > > > > CloudEvents-shaped. > > > > > > > - C) Fold the shared mapper into [1] itself (larger single PR). > > > > > > > > > > > > > > Unless someone prefers B or C, I will open the prerequisite PR > > > along > > > > the > > > > > > > lines of (1)-(4) early next week. > > > > > > > > > > > > > > Thanks, > > > > > > > Vignesh > > > > > > > > > > > > > > [1] https://github.com/apache/polaris/pull/5084 > > > > > > > > > > > > > > > > > > > > > > >
