I guess I replied to an old message in the middle of the thread :) My message still holds :)
Cheers, Dmitri. On Wed, Sep 23, 2026 at 7:53 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi EJ, > > Thanks for the meeting recap! > > All: > > Worth noting: PR [4115] also contains Management API changes that were not > anticipated initially and did not get an API change review until now. These > are small changes to add new grant types for metrics RBAC.. > > I've reviewed and approved PR [4115] in GH. > > [4115] https://github.com/apache/polaris/pull/4115 > > Cheers, > Dmitri. > > On Wed, Aug 12, 2026 at 5:21 PM EJ Wang <[email protected]> > wrote: > >> Hi folks, >> >> Notes from today's metrics architecture sync are in the usual doc: >> >> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.18iv1m7p6i6b >> >> Highlights: >> >> Module placement. core/ now holds the entity data model only. A new spi/ >> takes every other shared contract: feature SPIs, feature non-SPI >> contracts, >> substrate SPIs, substrate non-SPI contracts, and contract-required types. >> That settles the question PR #5204 reopened, by placing into a new SPI >> module. Note that it changes slightly towards the referenced WIP RFC [1] >> recommendations: what proposed to place in core/, now should go to spi/ >> except for the entity data model. >> >> Metrics PRs. #5068 merged. #5204 needs reshaping onto the split. #4115 and >> #4756 stay open with action on the author to follow up on review comments. >> >> [1] >> >> https://docs.google.com/document/d/1mj3vGpix9zTDTFFJKtWT5ZxI3PTGI1YNb5xFqyWrOkc/edit?tab=t.0 >> >> On Wed, Jul 15, 2026 at 11:01 AM Anand Kumar Sankaran via dev < >> [email protected]> wrote: >> >> > Thanks EJ. My plan is to introduce PR0 in a new branch / PR and >> repurpose >> > #4115 and #4756 for PR1 and PR2 to maintain all the valuable discussions >> > that happened there. >> > >> > From: EJ Wang <[email protected]> >> > Date: Wednesday, July 15, 2026 at 10:41 AM >> > To: [email protected] <[email protected]> >> > Subject: Re: Proposal for REST endpoints for table metrics and events >> > >> > Hi folks, Thanks again for the discussion today. I updated the sync doc >> > with notes from today's metrics sync: https: //urldefense. >> com/v3/__https: >> > //docs. google. >> > >> com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t. >> > >> 2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$ >> > >> > >> > Hi folks, >> > >> > Thanks again for the discussion today. I updated the sync doc with notes >> > from today's metrics sync: >> > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.2ahd7wze5f__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXJwAz1g0$ >> > >> > See highlights below: >> > >> > In previous meetings, EJ proposed a two prs framing to push forward the >> > metrics query work in polaris. Dmitri pointed out a subtle point that >> pr2 >> > is currently also trying to do the persistence related code placement >> > refactor. For the sake of straightforward review process, EJ, Dmitri and >> > Anand (the author) agreed to separate the persistence refactoring into >> pr0, >> > and reframing the pr scopes as follow: >> > >> > 1. >> > >> > Pr0: refactor irc metrics spi >> > 1. >> > >> > SPI should stay in core/ with metric type enveloped >> > 2. >> > >> > default spi impl, no-op, in extensions/ - default selection at >> > runtime/ >> > 3. >> > >> > move metrics persistence and correspondings to extensions/ as an >> > alternative SPI impl >> > 2. >> > >> > Pr1: rest api for query >> > >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXFCbEbRk$ >> > 1. >> > >> > rest api spec in extensions/ as optional spec >> > 2. >> > >> > SPI in extensions/ as optional spec spi >> > 3. >> > >> > default spi impl in extensions/ no-op >> > 4. >> > >> > handler related in runtime/ - thin adaptor from http shape to spi >> > shape >> > 3. >> > >> > Pr2: jdbc impl for query >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4756__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXRVMBzqE$ >> > 1. >> > >> > alternative spi impl in extensions/ -> query with jdbc >> > >> > >> > Thanks, >> > -ej >> > >> > On Wed, Jul 1, 2026 at 8:01 AM Alexandre Dutra <[email protected]> >> wrote: >> > >> > > Hi all, >> > > >> > > > While the current implementation stores metrics in the metastore, >> other >> > > deployments >> > > may choose a KV store, a dedicated metrics warehouse, or another >> backend >> > > entirely. In those cases, exposing a Polaris REST API over a specific >> > > schema feels like coupling the API to an implementation detail. >> > > >> > > I wouldn't say to an implementation detail, but rather to a particular >> > > setup / configuration. I think that's fine. If you don't store >> > > metrics, you don't read metrics. Similar situations will certainly >> > > happen in the future, e.g. if we ever introduce an API to read events, >> > > that would only work if events are being persisted in the first place. >> > > Again I think that's a fair trade-off. >> > > >> > > Thanks, >> > > Alex >> > > >> > > On Wed, Jul 1, 2026 at 7:23 AM Dmitri Bourlatchkov <[email protected]> >> > > wrote: >> > > > >> > > > Hi Yufei, >> > > > >> > > > From my POV, it is fine for Polaris to have multiple REST API >> surfaces >> > > with >> > > > different features and different degrees of "readiness". Each API >> can >> > > > evolve fairly independently. With proper modularization (which Anand >> > > did), >> > > > changes to the main Polaris code and these extra modules should >> > normally >> > > be >> > > > non-intrusive to each other. >> > > > >> > > > I assume Anand is going to be the first user of this API. However, >> if >> > we >> > > do >> > > > not merge it we'll never know whether other users are interested. I >> do >> > > not >> > > > think many end users read the dev ML, but they certainly read the >> docs, >> > > > code and release notes. >> > > > >> > > > There is no risk for Polaris as a project in exposing this >> experimental >> > > > API. If usage dwindles over time, we can remove it. If it gains a >> > > > considerable user / contributor base, we will promote it to a >> > > "first-class" >> > > > (non-beta) API. >> > > > >> > > > Cheers, >> > > > Dmitri. >> > > > >> > > > On Tue, Jun 30, 2026 at 7:53 PM Yufei Gu <[email protected]> >> wrote: >> > > > >> > > > > Hi all, >> > > > > >> > > > > Thanks for driving this work forward. Building this as a POC makes >> > > perfect >> > > > > sense if the goal is to validate the idea and gather feedback. >> Should >> > > we >> > > > > merge a POC into the main branch before agreeing on the overall >> > > direction? >> > > > > To me, that feels like a separate discussion from whether the >> > > > > implementation itself looks reasonable. >> > > > > >> > > > > One concern that I don't think has been fully addressed is >> whether we >> > > need >> > > > > to expose metrics consumption through a Polaris REST endpoint at >> all. >> > > > > >> > > > > My concern is less about the API itself and more about the >> boundary >> > > Polaris >> > > > > should own. The proposed REST API is tied to a particular >> persistence >> > > > > schema, but that schema is just one possible implementation. While >> > the >> > > > > current implementation stores metrics in the metastore, other >> > > deployments >> > > > > may choose a KV store, a dedicated metrics warehouse, or another >> > > backend >> > > > > entirely. In those cases, exposing a Polaris REST API over a >> specific >> > > > > schema feels like coupling the API to an implementation detail. >> > > > > >> > > > > Polaris isn't intended to become a full metrics system. I'd >> expect it >> > > to >> > > > > focus on producing metrics and let deployments choose the most >> > > appropriate >> > > > > storage and consumption model for their environment. >> > > > > >> > > > > I'm not opposed to exploring this direction, but we should first >> > align >> > > on >> > > > > whether exposing a REST API through Polaris is the right long term >> > > approach >> > > > > before merging it into the main branch. >> > > > > >> > > > > Thanks, >> > > > > Yufei >> > > > > >> > > > > Yufei >> > > > > >> > > > > >> > > > > On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov < >> > [email protected]> >> > > > > wrote: >> > > > > >> > > > > > Hi All, >> > > > > > >> > > > > > It looks like Anand addressed all action items in [4115]. >> > > > > > >> > > > > > Please review the latest state of the code. The PR LGTM and I >> > > approved in >> > > > > > GH from my side. >> > > > > > >> > > > > > [4115] >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$ >> > > > > > >> > > > > > Thanks, >> > > > > > Dmitri. >> > > > > > >> > > > > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov < >> > > [email protected]> >> > > > > > wrote: >> > > > > > >> > > > > > > Hi All, >> > > > > > > >> > > > > > > Recap of the discussion in today's Community Sync: >> > > > > > > >> > > > > > > * The modular approach for Metrics SPI(s) and implementation >> > seems >> > > to >> > > > > > have >> > > > > > > community consensus (please comment if you still have >> concerns) >> > > > > > > >> > > > > > > * What modules / feature flags are enabled in Apache Polaris >> > > images by >> > > > > > > default will be discussed later (when the feature is close to >> > > > > > completion). >> > > > > > > >> > > > > > > In any case, users will be able to turn the full feature >> on/off >> > at >> > > > > their >> > > > > > > discretion, regardless of the default flag values. >> > > > > > > >> > > > > > > * It is generally preferable to have smaller PRs and introduce >> > new >> > > > > > > features gradually to simplify reviews. >> > > > > > > >> > > > > > > What this means for the Metrics API, IMHO is: >> > > > > > > >> > > > > > > a) The scope of PR 4115 is probably ok as it stands right >> now. If >> > > > > someone >> > > > > > > wants to propose splitting it further, please comment. >> > > > > > > >> > > > > > > b) After 4115 is merged, Metrics Persistence PRs will follow, >> > with >> > > > > > smaller >> > > > > > > scope per PR as appropriate for the nature of the code >> changes. >> > > > > > > >> > > > > > > * The major concern was about exposing the new Metrics REST >> API >> > to >> > > end >> > > > > > > users. The nature of this concern was about avoiding the >> (false) >> > > > > > impression >> > > > > > > that the new API is a firm and adopted Polaris specification. >> > > > > > > >> > > > > > > I think this concern is reasonable. >> > > > > > > >> > > > > > > I propose that Anand update the documentation sections of PR >> 4115 >> > > to >> > > > > > > emphasise that the API is currently an experimental / POC API >> at >> > > this >> > > > > > time >> > > > > > > and is subject to change (including breaking changes). I >> believe >> > it >> > > > > > already >> > > > > > > has the "beta" label. So this request is mostly for doc >> changes >> > to >> > > > > > clarify >> > > > > > > user expectations, since the "beta" label alone may not be >> > > interpreted >> > > > > > the >> > > > > > > same way by all users. >> > > > > > > >> > > > > > > I hope this update will resolve all remaining concerns with >> 4115 >> > > and >> > > > > > allow >> > > > > > > for it to be merged so that the community will be able to >> proceed >> > > to >> > > > > > > follow-up PRs (as noted above). >> > > > > > > >> > > > > > > Please add/correct me if I missed anything relevant. >> > > > > > > >> > > > > > > Thanks, >> > > > > > > Dmitri. >> > > > > > > >> > > > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu < >> [email protected]> >> > > wrote: >> > > > > > > >> > > > > > >> I do not think we have agreement on the query REST APIs yet. >> > > > > > >> >> > > > > > >> In particular, the current query API shape seems highly >> coupled >> > > to the >> > > > > > >> *example[1]* metrics in the Iceberg REST spec. I do not think >> > > that is >> > > > > a >> > > > > > >> good way to design a REST API contract. For example, fields >> like >> > > > > > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes, >> > > > > > >> totalDataManifests, >> > > > > > >> totalDeleteManifests, and scannedDataManifests are very >> specific >> > > to >> > > > > one >> > > > > > >> style of scan metric. That makes the API look more like a >> direct >> > > > > > >> projection >> > > > > > >> of the current example payload than a general query model. >> > > > > > >> >> > > > > > >> My preference is that Polaris should not define or strongly >> > opine >> > > on >> > > > > the >> > > > > > >> Iceberg scan or commit metrics query REST API at this point. >> > > Polaris >> > > > > can >> > > > > > >> resolve and authorize the table, accept the metrics report, >> and >> > > > > delegate >> > > > > > >> it >> > > > > > >> to the configured reporting path. Durable storage, filtering, >> > > > > > dashboards, >> > > > > > >> and query APIs should remain downstream impl. choices unless >> and >> > > until >> > > > > > the >> > > > > > >> Iceberg community standardizes a query contract, which I >> doubt >> > > will >> > > > > > >> happen. >> > > > > > >> >> > > > > > >> So I do not think the intake path has to be blocked, but I >> would >> > > > > prefer >> > > > > > >> not >> > > > > > >> to merge a Polaris specific query API that may be hard to >> evolve >> > > > > later. >> > > > > > >> >> > > > > > >> 1. >> > > > > > >> >> > > > > > >> >> > > > > > >> > > > > >> > > >> > >> https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml*L4152__;Iw!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXCDsqYGo$ >> > > > > > >> >> > > > > > >> Yufei >> > > > > > >> >> > > > > > >> >> > > > > > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov < >> > > [email protected] >> > > > > > >> > > > > > >> wrote: >> > > > > > >> >> > > > > > >> > Hi All, >> > > > > > >> > >> > > > > > >> > So, what are the blockers for merging [4115], if any? >> > > > > > >> > >> > > > > > >> > [4115] >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$ >> > > > > > >> > >> > > > > > >> > Thanks, >> > > > > > >> > Dmitri. >> > > > > > >> > >> > > > > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang < >> > > > > > [email protected] >> > > > > > >> > >> > > > > > >> > wrote: >> > > > > > >> > >> > > > > > >> > > Hi folks, >> > > > > > >> > > >> > > > > > >> > > Thanks again for the discussion today. I updated the sync >> > doc >> > > with >> > > > > > >> notes >> > > > > > >> > > from today's metrics sync: >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrX83IZkm0$ >> > > > > > >> > > >> > > > > > >> > > A few highlights: >> > > > > > >> > > >> > > > > > >> > > - No objection to keeping the default metrics behavior >> > > small: >> > > > > > >> > > no-op/log-only is enough as the built-in default. >> > > > > > >> > > - Durable storage, event forwarding, external queues, >> > > > > dashboards, >> > > > > > >> and >> > > > > > >> > > custom filtering should be implementation choices >> behind >> > > the >> > > > > > >> metrics >> > > > > > >> > > reporting path. >> > > > > > >> > > - For PR #4115, the REST layer should >> > resolve/authz/accept >> > > the >> > > > > > >> report, >> > > > > > >> > > then delegate to the selected metrics implementation. >> It >> > > should >> > > > > > not >> > > > > > >> > own >> > > > > > >> > > durable storage semantics. >> > > > > > >> > > - The rough boundary we discussed is: spec owns the >> wire >> > > > > > contract, >> > > > > > >> > > runtime owns framework wiring, API/contract modules >> own >> > > > > > >> > provider-facing >> > > > > > >> > > contracts, and extensions own replaceable >> > implementations. >> > > > > > >> > > - OpenLineage has a similar path-naming question. We >> > should >> > > > > avoid >> > > > > > >> > > occupying a generic lineage path too early if we want >> > room >> > > for >> > > > > > >> other >> > > > > > >> > > lineage systems later. >> > > > > > >> > > >> > > > > > >> > > I believe the boundary rules previewed in sync meeting >> are >> > > useful >> > > > > > for >> > > > > > >> > > reviewing PR #4115, but it is not >> > > > > > >> > > a final module-layout proposal. >> > > > > > >> > > >> > > > > > >> > > For metrics specifically, I think the target remains: >> > > > > > >> > > runtime >> > > > > > >> > > -> metrics reporting/emitting contract >> > > > > > >> > > -> default implementation >> > > > > > >> > > -> lower implementation details, if needed >> > > > > > >> > > >> > > > > > >> > > That keeps the default simple, while allowing durable >> JDBC, >> > > > > > >> event-backed, >> > > > > > >> > > or external-queue implementations to be added separately. >> > > > > > >> > > >> > > > > > >> > > We also touched on the split between Iceberg metrics >> > emitting >> > > and >> > > > > > >> > > Polaris-owned metrics querying. The querying side can >> keep >> > > > > evolving, >> > > > > > >> > > especially for dashboard or generic-table use cases, but >> I >> > do >> > > not >> > > > > > >> think >> > > > > > >> > > that needs to block the intake path in this PR. >> > > > > > >> > > >> > > > > > >> > > Thanks, >> > > > > > >> > > -ej >> > > > > > >> > > >> > > > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu < >> > > [email protected]> >> > > > > > >> wrote: >> > > > > > >> > > >> > > > > > >> > > > Thanks for chiming in, EJ. Agreed that the default >> battery >> > > > > should >> > > > > > >> stay >> > > > > > >> > > > small. I'm leaning toward not including metrics >> > persistence >> > > in >> > > > > the >> > > > > > >> > > default >> > > > > > >> > > > battery. >> > > > > > >> > > > >> > > > > > >> > > > Yufei >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang < >> > > > > > >> > [email protected]> >> > > > > > >> > > > wrote: >> > > > > > >> > > > >> > > > > > >> > > > > Hi Yufei, >> > > > > > >> > > > > >> > > > > > >> > > > > Thanks for connecting this back to the REST endpoint >> > > > > proposal. I >> > > > > > >> > > replied >> > > > > > >> > > > > with the fuller version on the event-forwarding >> thread, >> > > but >> > > > > > >> wanted to >> > > > > > >> > > add >> > > > > > >> > > > > the shorter version here too. >> > > > > > >> > > > > >> > > > > > >> > > > > I agree that metrics reporting/emitting is the right >> > > > > conceptual >> > > > > > >> > > boundary, >> > > > > > >> > > > > and I also think the event/listener path is a good >> > > > > > implementation >> > > > > > >> > > > direction >> > > > > > >> > > > > to explore. The distinction I want to keep clear is >> > > default >> > > > > > >> battery >> > > > > > >> > vs >> > > > > > >> > > > > extension implementation. >> > > > > > >> > > > > >> > > > > > >> > > > > For the REST/API side, I would keep the semantics >> > narrow: >> > > > > > Polaris >> > > > > > >> > > > resolves >> > > > > > >> > > > > and authorizes the table, accepts the Iceberg >> > scan/commit >> > > > > > metrics >> > > > > > >> > > report >> > > > > > >> > > > > into the configured reporting/emitting path, and >> returns >> > > 204. >> > > > > > That >> > > > > > >> > 204 >> > > > > > >> > > > > should mean Polaris accepted the report into the >> > ingestion >> > > > > path; >> > > > > > >> it >> > > > > > >> > > > should >> > > > > > >> > > > > not imply durable storage. >> > > > > > >> > > > > >> > > > > > >> > > > > Durable storage, event forwarding, external telemetry >> > > routing, >> > > > > > >> > > filtering, >> > > > > > >> > > > > and retention should sit behind that >> reporting/emitting >> > > > > boundary >> > > > > > >> as >> > > > > > >> > > > > implementation-layer behavior. A durable JDBC path >> can >> > be >> > > one >> > > > > > >> named >> > > > > > >> > > > > extension implementation. An event/listener >> forwarding >> > > path >> > > > > can >> > > > > > be >> > > > > > >> > > > another >> > > > > > >> > > > > named extension implementation. >> > > > > > >> > > > > >> > > > > > >> > > > > So I am +1 on the event-forwarding idea as a >> non-default >> > > > > > extension >> > > > > > >> > > > > implementation. I just would not make it the default >> > > battery >> > > > > or >> > > > > > >> the >> > > > > > >> > > core >> > > > > > >> > > > > API shape. The default battery can stay small and >> safe, >> > > while >> > > > > > >> > > deployments >> > > > > > >> > > > > that need forwarding, dashboards, or durable storage >> can >> > > opt >> > > > > > into >> > > > > > >> the >> > > > > > >> > > > > implementation that matches their operational model. >> > > > > > >> > > > > >> > > > > > >> > > > > Thanks, >> > > > > > >> > > > > -ej >> > > > > > >> > > > > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu < >> > > > > [email protected] >> > > > > > > >> > > > > > >> > > wrote: >> > > > > > >> > > > > >> > > > > > >> > > > > > Thanks EJ. I agree that the reporting/emitting >> > boundary >> > > > > feels >> > > > > > >> like >> > > > > > >> > > the >> > > > > > >> > > > > > right SPI boundary, but I wonder if we can simplify >> > this >> > > > > even >> > > > > > >> > > further. >> > > > > > >> > > > > > >> > > > > > >> > > > > > Iceberg scan and commit metrics look very similar >> to >> > > events >> > > > > to >> > > > > > >> me. >> > > > > > >> > > They >> > > > > > >> > > > > are >> > > > > > >> > > > > > append only, asynchronously consumed, often >> forwarded >> > to >> > > > > > >> external >> > > > > > >> > > > > systems, >> > > > > > >> > > > > > and have similar retention and cleanup >> requirements. >> > > More >> > > > > > >> details >> > > > > > >> > can >> > > > > > >> > > > be >> > > > > > >> > > > > > found in this thread [1]. In that case, Polaris may >> > only >> > > > > need >> > > > > > to >> > > > > > >> > > accept >> > > > > > >> > > > > the >> > > > > > >> > > > > > report and emit an event through the existing event >> > > > > framework. >> > > > > > >> > > > Filtering, >> > > > > > >> > > > > > async delivery, custom sinks, and retention >> mechanisms >> > > could >> > > > > > >> then >> > > > > > >> > be >> > > > > > >> > > > > shared >> > > > > > >> > > > > > instead of introducing a separate metrics specific >> > > extension >> > > > > > >> layer >> > > > > > >> > > and >> > > > > > >> > > > > SPI. >> > > > > > >> > > > > > >> > > > > > >> > > > > > The main requirement I still see is querying >> metrics >> > > through >> > > > > > >> > Polaris. >> > > > > > >> > > > If >> > > > > > >> > > > > we >> > > > > > >> > > > > > want that battery included experience, we could >> work >> > on >> > > a >> > > > > > >> > persistence >> > > > > > >> > > > > > listener implementation that selectively saves >> metrics >> > > and >> > > > > > >> expose >> > > > > > >> > it >> > > > > > >> > > > via >> > > > > > >> > > > > > the IRC event endpoint which seems pretty close in >> > > Iceberg >> > > > > > >> > > > > > community. Another option I like more is to allow >> > users >> > > to >> > > > > > >> develop >> > > > > > >> > > > their >> > > > > > >> > > > > > own listener, so they can persist scan/commit >> metrics >> > > to any >> > > > > > >> > systems >> > > > > > >> > > > they >> > > > > > >> > > > > > prefer. In general, that feels like an >> implementation >> > > choice >> > > > > > on >> > > > > > >> top >> > > > > > >> > > of >> > > > > > >> > > > > the >> > > > > > >> > > > > > event framework rather than something that needs to >> > > shape >> > > > > the >> > > > > > >> core >> > > > > > >> > > > > > architecture. >> > > > > > >> > > > > > >> > > > > > >> > > > > > 1. >> > > > > > >> > >> > > >> > >> https://urldefense.com/v3/__https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXGcGDgks$ >> > > > > > >> > > > > > >> > > > > > >> > > > > > Thanks, >> > > > > > >> > > > > > Yufei >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang < >> > > > > > >> > > [email protected] >> > > > > > >> > > > > >> > > > > > >> > > > > > wrote: >> > > > > > >> > > > > > >> > > > > > >> > > > > > > Good point. I agree the existing >> > > PolarisMetricsReporter is >> > > > > > >> > already >> > > > > > >> > > > very >> > > > > > >> > > > > > > close to the right conceptual boundary. I am not >> > > > > proposing a >> > > > > > >> > second >> > > > > > >> > > > > > > parallel reporter concept. The distinction I am >> > > trying to >> > > > > > >> make is >> > > > > > >> > > > > > narrower: >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > Current state: >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > - PolarisMetricsReporter is the existing >> metrics >> > > > > > reporting >> > > > > > >> > hook. >> > > > > > >> > > > > > > - But it currently lives under >> runtime/service. >> > > > > > >> > > > > > > - Its contract is documented around >> Quarkus/CDI >> > > > > discovery >> > > > > > >> and >> > > > > > >> > > > > > > @Identifier >> > > > > > >> > > > > > > selection. >> > > > > > >> > > > > > > - The default implementation is log-only. >> > > > > > >> > > > > > > - The durable implementation writes through >> > > > > > >> > > PolarisMetricsManager >> > > > > > >> > > > > and >> > > > > > >> > > > > > > MetricsPersistence. >> > > > > > >> > > > > > > - MetricsPersistence is currently inherited by >> > > > > > >> > BasePersistence. >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > My proposal: >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > - Keep the reporting/emitting boundary as the >> > SPI. >> > > > > > >> > > > > > > - Revise that contract in a framework-agnostic >> > > optional >> > > > > > >> > Iceberg >> > > > > > >> > > > > > *metrics >> > > > > > >> > > > > > > extension API layer*. >> > > > > > >> > > > > > > - Keep runtime/service responsible for REST >> > > ingestion, >> > > > > > >> > > table/authz >> > > > > > >> > > > > > > validation, and runtime wiring. >> > > > > > >> > > > > > > - Keep the battery implementation log-only in >> the >> > > > > metrics >> > > > > > >> > > > extension >> > > > > > >> > > > > > API >> > > > > > >> > > > > > > layer, *not under runtime/service*. >> > > > > > >> > > > > > > - Treat durable JDBC metrics as one >> > implementation >> > > of >> > > > > the >> > > > > > >> > > > reporting >> > > > > > >> > > > > > SPI. >> > > > > > >> > > > > > > - Decompose the MetricsPersistence and >> > > BasePersistence >> > > > > > >> > coupling >> > > > > > >> > > as >> > > > > > >> > > > > > > durable implementation detail. >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > So the difference is not “new reporter concept vs >> > old >> > > > > > reporter >> > > > > > >> > > > > concept.” >> > > > > > >> > > > > > > but: >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > - PolarisMetricsReporter today = >> runtime/service >> > > > > > CDI-shaped >> > > > > > >> > > hook. >> > > > > > >> > > > > > > - Target reporting SPI = same conceptual >> > boundary, >> > > but >> > > > > > >> placed >> > > > > > >> > in >> > > > > > >> > > > the >> > > > > > >> > > > > > > right extension API layer and kept >> > > framework-agnostic. >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > For the current metrics PR, I think the minimal >> > > > > > architectural >> > > > > > >> > > cleanup >> > > > > > >> > > > > > > before merge is to *put the reporting SPI >> contract >> > > and the >> > > > > > >> > battery >> > > > > > >> > > > > > default >> > > > > > >> > > > > > > in the right place*. The durable JDBC >> > implementation, >> > > > > > >> > event-backed >> > > > > > >> > > > > async >> > > > > > >> > > > > > > implementation, external queue support, and >> deeper >> > > > > > persistence >> > > > > > >> > > > cleanup >> > > > > > >> > > > > > can >> > > > > > >> > > > > > > continue as follow-ups, as long as we do not lock >> > the >> > > SPI >> > > > > > >> > boundary >> > > > > > >> > > to >> > > > > > >> > > > > > > BasePersistence or to runtime/service wiring. >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > -ej >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri >> Bourlatchkov < >> > > > > > >> > > [email protected] >> > > > > > >> > > > > >> > > > > > >> > > > > > > wrote: >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > > Hi EJ, >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > Thanks for the recap / summary! >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > > Do folks agree that the stable SPI boundary >> > > should be >> > > > > > >> > > > > > > > metrics reporting/emitting, not metrics >> > persistence? >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > SGTM. >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > > Does an optional Iceberg metrics extension >> API >> > > layer >> > > > > > sound >> > > > > > >> > like >> > > > > > >> > > > the >> > > > > > >> > > > > > > right >> > > > > > >> > > > > > > > home for this SPI? >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > How is that different from current >> > > > > PolarisMetricsReporter? >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > > Should the current durable metrics work be >> > > reframed >> > > > > as a >> > > > > > >> > > durable >> > > > > > >> > > > > > > > JDBC reference implementation of that SPI? >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > SGTM. >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > > What is the smallest PR sequence to get there >> > > without >> > > > > > >> > blocking >> > > > > > >> > > > > > > > the current metrics work unnecessarily? >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > Let's get >> > > >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$ >> > > > > > >> merged >> > > > > > >> > > > first. >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > Then, I'd propose isolating the Metrics schema >> > from >> > > the >> > > > > > >> > MetaStore >> > > > > > >> > > > > > schema. >> > > > > > >> > > > > > > > This will probably have some ripple effect into >> > the >> > > > > > >> bootstrap >> > > > > > >> > > > > > workflows, >> > > > > > >> > > > > > > so >> > > > > > >> > > > > > > > it's not a trivial change. >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > Then, let's reassess. >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > Cheers, >> > > > > > >> > > > > > > > Dmitri. >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang < >> > > > > > >> > > > > [email protected] >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > > wrote: >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > > Hi folks, >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > Thanks again for the discussion today. I >> updated >> > > the >> > > > > > sync >> > > > > > >> doc >> > > > > > >> > > > with >> > > > > > >> > > > > > > notes >> > > > > > >> > > > > > > > > from the third metrics architecture sync: >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXrqCL-QE$ >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > A few highlights from today’s discussion: >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > - We clarified that Iceberg metrics >> reporting >> > > can >> > > > > be >> > > > > > >> > > > interpreted >> > > > > > >> > > > > > > > either >> > > > > > >> > > > > > > > > as sync or async handling from Polaris’s >> > > > > perspective, >> > > > > > >> so >> > > > > > >> > > > Polaris >> > > > > > >> > > > > > > > should >> > > > > > >> > > > > > > > > stay flexible as a platform instead of >> baking >> > > one >> > > > > > >> handling >> > > > > > >> > > > model >> > > > > > >> > > > > > > into >> > > > > > >> > > > > > > > > the >> > > > > > >> > > > > > > > > REST API behavior. >> > > > > > >> > > > > > > > > - We aligned that the built-in (battery) >> > > behavior >> > > > > for >> > > > > > >> > > metrics >> > > > > > >> > > > > > > emitting >> > > > > > >> > > > > > > > > can stay simple: no-op/log-only is enough >> as >> > > the >> > > > > > >> default. >> > > > > > >> > > > > > > > > - Durable metrics persistence should be >> > > treated as >> > > > > an >> > > > > > >> > > > > > implementation >> > > > > > >> > > > > > > > of >> > > > > > >> > > > > > > > > the metrics reporting path, not as the >> core >> > SPI >> > > > > > >> boundary. >> > > > > > >> > > > > > > > > - The existing durable metrics work can be >> > > reviewed >> > > > > > as >> > > > > > >> a >> > > > > > >> > > > > reference >> > > > > > >> > > > > > > > > implementation of the reporting SPI, with >> > > > > > >> > > persistence-related >> > > > > > >> > > > > > logic >> > > > > > >> > > > > > > > kept >> > > > > > >> > > > > > > > > self-contained under a metrics durable >> > > > > implementation >> > > > > > >> > module >> > > > > > >> > > > > > rather >> > > > > > >> > > > > > > > than >> > > > > > >> > > > > > > > > scattered through core entity persistence. >> > > > > > >> > > > > > > > > - Dashboard/insights remains a real use >> case, >> > > but >> > > > > we >> > > > > > >> > agreed >> > > > > > >> > > to >> > > > > > >> > > > > > keep >> > > > > > >> > > > > > > it >> > > > > > >> > > > > > > > > separate from the core metrics intake >> > > discussion >> > > > > for >> > > > > > >> now. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > I also did a quick source check after the >> > meeting >> > > to >> > > > > > make >> > > > > > >> > sure >> > > > > > >> > > we >> > > > > > >> > > > > are >> > > > > > >> > > > > > > > > describing the current state accurately. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > Current state: >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > - Polaris already has a metrics reporting >> > hook: >> > > > > > >> > > > > > > > PolarisMetricsReporter. >> > > > > > >> > > > > > > > > - The default implementation is >> > > > > > DefaultMetricsReporter, >> > > > > > >> > > > selected >> > > > > > >> > > > > > by >> > > > > > >> > > > > > > > > >> > > polaris.iceberg-metrics.reporting.type=default. It >> > > > > is >> > > > > > >> > > > log-only, >> > > > > > >> > > > > > > > > effectively quiet unless metrics logging >> is >> > > > > enabled. >> > > > > > >> > > > > > > > > - There is also a >> PersistingMetricsReporter, >> > > > > selected >> > > > > > >> by >> > > > > > >> > > > > > > > > >> > > polaris.iceberg-metrics.reporting.type=persisting, >> > > > > > >> which >> > > > > > >> > > > > converts >> > > > > > >> > > > > > > > > Iceberg scan/commit reports into Polaris >> > > metrics >> > > > > > >> records >> > > > > > >> > and >> > > > > > >> > > > > > writes >> > > > > > >> > > > > > > > > through >> > > > > > >> > > > > > > > > PolarisMetricsManager -> >> MetricsPersistence. >> > > > > > >> > > > > > > > > - MetricsPersistence currently exists in >> the >> > > > > > >> persistence >> > > > > > >> > > layer >> > > > > > >> > > > > and >> > > > > > >> > > > > > > is >> > > > > > >> > > > > > > > > inherited by BasePersistence. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > My read from the discussion is that the >> target >> > > > > boundary >> > > > > > >> > should >> > > > > > >> > > > not >> > > > > > >> > > > > be >> > > > > > >> > > > > > > > > MetricsPersistence as inherited by >> > > BasePersistence. >> > > > > That >> > > > > > >> path >> > > > > > >> > > > > should >> > > > > > >> > > > > > be >> > > > > > >> > > > > > > > > decomposed as durable implementation detail >> > (taken >> > > > > care >> > > > > > >> of by >> > > > > > >> > > > > > > > > >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4397__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuu5jefU$ >> ). >> > The >> > > > > > stable >> > > > > > >> > > > extension >> > > > > > >> > > > > > > point >> > > > > > >> > > > > > > > > should instead be the metrics >> reporting/emitting >> > > > > > boundary. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > Concretely, I think the next proposal should >> be >> > > shaped >> > > > > > >> like >> > > > > > >> > > this: >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > 1. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > Define the proper metrics reporting SPI >> at an >> > > > > > optional >> > > > > > >> > > Iceberg >> > > > > > >> > > > > > > metrics >> > > > > > >> > > > > > > > > extension API layer. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > Example direction: >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > public interface IcebergMetricsReporter { >> > > > > > >> > > > > > > > > void >> > reportMetric(IcebergMetricsReportContext >> > > > > > >> context, >> > > > > > >> > > > > > > > > MetricsReport report); >> > > > > > >> > > > > > > > > } >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > The context would carry the small >> > > Polaris-resolved >> > > > > > >> > envelope: >> > > > > > >> > > > > > > > > catalog/table identity, report type, >> received >> > > > > > >> timestamp, >> > > > > > >> > and >> > > > > > >> > > > > > > > > request/trace >> > > > > > >> > > > > > > > > context if available. The raw Iceberg >> > > MetricsReport >> > > > > > >> > remains >> > > > > > >> > > > the >> > > > > > >> > > > > > > > payload. >> > > > > > >> > > > > > > > > 2. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > Keep runtime/service as the REST ingestion >> > and >> > > > > wiring >> > > > > > >> > layer. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > The REST handler still resolves the table >> and >> > > > > > performs >> > > > > > >> > authz >> > > > > > >> > > > > > before >> > > > > > >> > > > > > > > > accepting the report. After that, it calls >> > the >> > > > > > selected >> > > > > > >> > > > > reporting >> > > > > > >> > > > > > > > > implementation. >> > > > > > >> > > > > > > > > 3. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > Keep the battery default no-op/log-only. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > This preserves an out-of-box safe default >> and >> > > > > avoids >> > > > > > >> > > > requiring a >> > > > > > >> > > > > > > > durable >> > > > > > >> > > > > > > > > metrics store for every Polaris >> deployment. >> > > > > > >> > > > > > > > > 4. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > Move durable JDBC metrics into a >> > self-contained >> > > > > > >> > > implementation >> > > > > > >> > > > > of >> > > > > > >> > > > > > > the >> > > > > > >> > > > > > > > > reporting SPI. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > That implementation can own its schema, >> > > bootstrap, >> > > > > > >> > > retention, >> > > > > > >> > > > > and >> > > > > > >> > > > > > > read >> > > > > > >> > > > > > > > > API support. It should not define the core >> > > > > reporting >> > > > > > >> SPI >> > > > > > >> > > > > boundary, >> > > > > > >> > > > > > > and >> > > > > > >> > > > > > > > > it >> > > > > > >> > > > > > > > > should not require metrics persistence to >> > > remain >> > > > > > >> inherited >> > > > > > >> > > > from >> > > > > > >> > > > > > > > > BasePersistence. >> > > > > > >> > > > > > > > > 5. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > Treat async/event-backed handling as >> another >> > > > > > >> > implementation >> > > > > > >> > > of >> > > > > > >> > > > > the >> > > > > > >> > > > > > > > same >> > > > > > >> > > > > > > > > reporting SPI. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > For example, an event-backed reporter >> could >> > > > > enqueue a >> > > > > > >> > > metrics >> > > > > > >> > > > > > event >> > > > > > >> > > > > > > > and >> > > > > > >> > > > > > > > > let listeners handle durable storage or >> other >> > > > > sinks. >> > > > > > >> If we >> > > > > > >> > > > later >> > > > > > >> > > > > > > need >> > > > > > >> > > > > > > > a >> > > > > > >> > > > > > > > > replaceable queue engine, that seems like >> a >> > > shared >> > > > > > >> > > > event/metrics >> > > > > > >> > > > > > > > > substrate >> > > > > > >> > > > > > > > > topic rather than a metrics-only >> requirement. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > This framing lets us keep the REST metrics >> > > endpoint >> > > > > > >> simple, >> > > > > > >> > > > > preserve >> > > > > > >> > > > > > > the >> > > > > > >> > > > > > > > > current default behavior, support durable >> > metrics >> > > > > users, >> > > > > > >> and >> > > > > > >> > > > still >> > > > > > >> > > > > > > leave >> > > > > > >> > > > > > > > > room for async/event-backed or >> > > external-queue-based >> > > > > > >> > > > > implementations. >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > I think the main follow-up questions are: >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > - Do folks agree that the stable SPI >> boundary >> > > > > should >> > > > > > be >> > > > > > >> > > > metrics >> > > > > > >> > > > > > > > > reporting/emitting, not metrics >> persistence? >> > > > > > >> > > > > > > > > - Does an optional Iceberg metrics >> extension >> > > API >> > > > > > layer >> > > > > > >> > sound >> > > > > > >> > > > > like >> > > > > > >> > > > > > > the >> > > > > > >> > > > > > > > > right home for this SPI? >> > > > > > >> > > > > > > > > - Should the current durable metrics work >> be >> > > > > reframed >> > > > > > >> as a >> > > > > > >> > > > > durable >> > > > > > >> > > > > > > > JDBC >> > > > > > >> > > > > > > > > reference implementation of that SPI? >> > > > > > >> > > > > > > > > - What is the smallest PR sequence to get >> > there >> > > > > > without >> > > > > > >> > > > blocking >> > > > > > >> > > > > > the >> > > > > > >> > > > > > > > > current metrics work unnecessarily? >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > Thanks, >> > > > > > >> > > > > > > > > -ej >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri >> > > Bourlatchkov < >> > > > > > >> > > > > > [email protected]> >> > > > > > >> > > > > > > > > wrote: >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > > Hi JB, >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > Could you set up another meeting, please? >> Same >> > > time >> > > > > on >> > > > > > >> > > > Wednesday >> > > > > > >> > > > > as >> > > > > > >> > > > > > > > last >> > > > > > >> > > > > > > > > > time... I hope it works for everyone. >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > Cheers, >> > > > > > >> > > > > > > > > > Dmitri. >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu < >> > > > > > >> > > [email protected] >> > > > > > >> > > > > >> > > > > > >> > > > > > > wrote: >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > > +1 on another sync call next week. >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > > Yufei >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri >> > > > > Bourlatchkov >> > > > > > < >> > > > > > >> > > > > > > > [email protected]> >> > > > > > >> > > > > > > > > > > wrote: >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > Hi All, >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > WDYT about another sync call next week? >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > Thanks, >> > > > > > >> > > > > > > > > > > > Dmitri. >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri >> > > > > > Bourlatchkov < >> > > > > > >> > > > > > > > [email protected] >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > > > wrote: >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > Hi EJ, >> > > > > > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > Thanks for the summary! It covers >> what >> > we >> > > > > > >> discussed >> > > > > > >> > in >> > > > > > >> > > > the >> > > > > > >> > > > > > > > meeting >> > > > > > >> > > > > > > > > > very >> > > > > > >> > > > > > > > > > > > > well, IMHO. >> > > > > > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > Looking forward to concrete PRs :) >> > > > > > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > Cheers, >> > > > > > >> > > > > > > > > > > > > Dmitri. >> > > > > > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ >> Wang < >> > > > > > >> > > > > > > > > > [email protected] >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > wrote: >> > > > > > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > >> Hi folks, >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> We had a community sync earlier, >> thanks >> > > JB >> > > > > for >> > > > > > >> > > > scheduling >> > > > > > >> > > > > > it. >> > > > > > >> > > > > > > > > Notes >> > > > > > >> > > > > > > > > > > from >> > > > > > >> > > > > > > > > > > > >> the first metrics architecture sync >> > (May >> > > 6, >> > > > > > >> 10-11am >> > > > > > >> > > PT). >> > > > > > >> > > > > > > > > Discussion >> > > > > > >> > > > > > > > > > > doc >> > > > > > >> > > > > > > > > > > > >> with per-section status: >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXgIddeUs$ >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> *The meeting covered both topics >> from >> > the >> > > > > doc. >> > > > > > >> > > > > > Direction-level >> > > > > > >> > > > > > > > > > > alignment >> > > > > > >> > > > > > > > > > > > >> was reached on the headline pieces; >> > > details >> > > > > > >> remain >> > > > > > >> > for >> > > > > > >> > > > PR >> > > > > > >> > > > > > > review >> > > > > > >> > > > > > > > > or >> > > > > > >> > > > > > > > > > > > >> follow-up sessions.* >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema >> redesign* >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment on >> consolidating >> > > > > per-type >> > > > > > >> > tables >> > > > > > >> > > > > > > > > > > > >> (scan_metrics_report, >> > > > > > >> > > > > > > > > > > > >> commit_metrics_report) into a single >> > > > > > >> metrics_report >> > > > > > >> > > > table. >> > > > > > >> > > > > > The >> > > > > > >> > > > > > > > > > > > motivating >> > > > > > >> > > > > > > > > > > > >> cost is the surface area added by >> every >> > > new >> > > > > > >> metric >> > > > > > >> > > type >> > > > > > >> > > > > > today: >> > > > > > >> > > > > > > > new >> > > > > > >> > > > > > > > > > > > table, >> > > > > > >> > > > > > > > > > > > >> SPI method, record class, model, >> > > converter, >> > > > > > >> schema >> > > > > > >> > > > > > migration. >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> Most schema details are deferred to >> the >> > > > > schema >> > > > > > >> PR. A >> > > > > > >> > > few >> > > > > > >> > > > > > > > specific >> > > > > > >> > > > > > > > > > > points >> > > > > > >> > > > > > > > > > > > >> came up: >> > > > > > >> > > > > > > > > > > > >> • metric_schema_version: Yufei >> > prefers >> > > > > > dropping >> > > > > > >> > it, >> > > > > > >> > > > > since >> > > > > > >> > > > > > > > there >> > > > > > >> > > > > > > > > is >> > > > > > >> > > > > > > > > > > no >> > > > > > >> > > > > > > > > > > > >> spec-level concept of metrics >> > versioning >> > > > > today >> > > > > > >> and >> > > > > > >> > it >> > > > > > >> > > is >> > > > > > >> > > > > > hard >> > > > > > >> > > > > > > to >> > > > > > >> > > > > > > > > > > define >> > > > > > >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping >> > it, >> > > > > given >> > > > > > >> IRC >> > > > > > >> > v2 >> > > > > > >> > > is >> > > > > > >> > > > > > > coming >> > > > > > >> > > > > > > > > and >> > > > > > >> > > > > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> schema should be considered against >> its >> > > > > likely >> > > > > > >> > shape; >> > > > > > >> > > > > Robert >> > > > > > >> > > > > > > > also >> > > > > > >> > > > > > > > > > > raised >> > > > > > >> > > > > > > > > > > > >> how to differentiate various payload >> > > formats >> > > > > if >> > > > > > >> any. >> > > > > > >> > > > EJ's >> > > > > > >> > > > > > read >> > > > > > >> > > > > > > > is >> > > > > > >> > > > > > > > > > that >> > > > > > >> > > > > > > > > > > > >> this >> > > > > > >> > > > > > > > > > > > >> is a two-way-door decision. We can >> > start >> > > > > > without >> > > > > > >> the >> > > > > > >> > > > > field, >> > > > > > >> > > > > > > and >> > > > > > >> > > > > > > > if >> > > > > > >> > > > > > > > > > IRC >> > > > > > >> > > > > > > > > > > > v2 >> > > > > > >> > > > > > > > > > > > >> changes the shape we would likely >> roll >> > a >> > > > > > >> > corresponding >> > > > > > >> > > > new >> > > > > > >> > > > > > > > schema >> > > > > > >> > > > > > > > > > > > anyway, >> > > > > > >> > > > > > > > > > > > >> which is not particularly costly. >> > > > > > >> > > > > > > > > > > > >> • Payload format: Robert pointed >> out >> > > that >> > > > > > >> future >> > > > > > >> > > > formats >> > > > > > >> > > > > > > > beyond >> > > > > > >> > > > > > > > > > JSON >> > > > > > >> > > > > > > > > > > > may >> > > > > > >> > > > > > > > > > > > >> be worth supporting. The exact >> shape is >> > > > > > deferred >> > > > > > >> to >> > > > > > >> > > the >> > > > > > >> > > > > > schema >> > > > > > >> > > > > > > > > > > > discussion. >> > > > > > >> > > > > > > > > > > > >> • Partition strategy: Anand >> suggested >> > > > > monthly >> > > > > > >> > > > > partitioning >> > > > > > >> > > > > > > > based >> > > > > > >> > > > > > > > > > on >> > > > > > >> > > > > > > > > > > > his >> > > > > > >> > > > > > > > > > > > >> experience as potentially helpful at >> > > scale. >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion >> and >> > > > > storage >> > > > > > >> > belong* >> > > > > > >> > > > > > > > > > > > >> Idea-level alignment that metrics >> > should >> > > be a >> > > > > > >> > > separated >> > > > > > >> > > > > SPI >> > > > > > >> > > > > > > from >> > > > > > >> > > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> entity >> > > > > > >> > > > > > > > > > > > >> persistence stack. Two reasons >> > surfaced: >> > > (a) >> > > > > > >> > workloads >> > > > > > >> > > > and >> > > > > > >> > > > > > > > > > capability >> > > > > > >> > > > > > > > > > > > >> requirements diverge enough that >> > coupling >> > > > > them >> > > > > > >> > creates >> > > > > > >> > > > > > > > artificial >> > > > > > >> > > > > > > > > > > > >> constraints, and (b) admin >> experience >> > > > > improves >> > > > > > >> when >> > > > > > >> > > > > metrics >> > > > > > >> > > > > > > has >> > > > > > >> > > > > > > > > its >> > > > > > >> > > > > > > > > > > own >> > > > > > >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle. >> > > Dmitri >> > > > > > noted >> > > > > > >> > > > Polaris >> > > > > > >> > > > > > > being >> > > > > > >> > > > > > > > a >> > > > > > >> > > > > > > > > > > > platform >> > > > > > >> > > > > > > > > > > > >> should have the flexibility to >> support >> > > > > > different >> > > > > > >> > > > > persistence >> > > > > > >> > > > > > > > > > backends >> > > > > > >> > > > > > > > > > > > per >> > > > > > >> > > > > > > > > > > > >> concern, and pointed to a concrete >> next >> > > step >> > > > > of >> > > > > > >> > > > separating >> > > > > > >> > > > > > the >> > > > > > >> > > > > > > > > JDBC >> > > > > > >> > > > > > > > > > > > >> bootstrap for metrics from the >> > metastore >> > > > > > >> bootstrap. >> > > > > > >> > > > Robert >> > > > > > >> > > > > > > > > proposed >> > > > > > >> > > > > > > > > > an >> > > > > > >> > > > > > > > > > > > >> additional UX extension: detect an >> > > > > > unbootstrapped >> > > > > > >> > > > metrics >> > > > > > >> > > > > > > store >> > > > > > >> > > > > > > > on >> > > > > > >> > > > > > > > > > > first >> > > > > > >> > > > > > > > > > > > >> use and auto-bootstrap rather than >> > > requiring >> > > > > an >> > > > > > >> > > explicit >> > > > > > >> > > > > > > manual >> > > > > > >> > > > > > > > > > > > bootstrap >> > > > > > >> > > > > > > > > > > > >> step. >> > > > > > >> > > > > > > > > > > > >> The meeting also confirmed that >> Polaris >> > > > > metrics >> > > > > > >> can >> > > > > > >> > > > start >> > > > > > >> > > > > > > small >> > > > > > >> > > > > > > > > and >> > > > > > >> > > > > > > > > > > stay >> > > > > > >> > > > > > > > > > > > >> Iceberg-focused. Naming and >> persistence >> > > > > schema >> > > > > > >> can >> > > > > > >> > > lean >> > > > > > >> > > > > > > > > > > > Iceberg-specific. >> > > > > > >> > > > > > > > > > > > >> If a future expansion to >> generic-table >> > > > > metrics >> > > > > > or >> > > > > > >> > > > > > operational >> > > > > > >> > > > > > > > > > metrics >> > > > > > >> > > > > > > > > > > > >> arrives, an abstraction layer can be >> > > built on >> > > > > > >> top of >> > > > > > >> > > the >> > > > > > >> > > > > > > Iceberg >> > > > > > >> > > > > > > > > > > metrics >> > > > > > >> > > > > > > > > > > > >> reporter at that point. Robert >> remains >> > > on the >> > > > > > >> fence >> > > > > > >> > > and >> > > > > > >> > > > > > would >> > > > > > >> > > > > > > > > prefer >> > > > > > >> > > > > > > > > > > > >> something more generic but did not >> > block >> > > the >> > > > > > >> > > direction; >> > > > > > >> > > > > > > Dmitri's >> > > > > > >> > > > > > > > > > read >> > > > > > >> > > > > > > > > > > > was >> > > > > > >> > > > > > > > > > > > >> that the proposed framework already >> has >> > > > > enough >> > > > > > >> > > > flexibility >> > > > > > >> > > > > > to >> > > > > > >> > > > > > > > > absorb >> > > > > > >> > > > > > > > > > > > >> future >> > > > > > >> > > > > > > > > > > > >> expansion. >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> The Trade-offs and Proposed >> structure >> > > > > sections >> > > > > > in >> > > > > > >> > the >> > > > > > >> > > > doc >> > > > > > >> > > > > > were >> > > > > > >> > > > > > > > not >> > > > > > >> > > > > > > > > > > > >> reviewed >> > > > > > >> > > > > > > > > > > > >> in detail. They remain open for >> either >> > > the >> > > > > next >> > > > > > >> sync >> > > > > > >> > > or >> > > > > > >> > > > PR >> > > > > > >> > > > > > > > review. >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> *Cross-cutting alignment — >> > > battery-included >> > > > > > plus >> > > > > > >> > > > > pluggable* >> > > > > > >> > > > > > > > > > > > >> A common philosophy emerged from the >> > > > > > discussion. >> > > > > > >> EJ >> > > > > > >> > > > > > summarized >> > > > > > >> > > > > > > > it >> > > > > > >> > > > > > > > > > as: >> > > > > > >> > > > > > > > > > > > >> Polaris should provide a >> > > battery-included UX >> > > > > > for >> > > > > > >> > > > beginners >> > > > > > >> > > > > > and >> > > > > > >> > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> flexibility for advanced users to >> swap >> > > the >> > > > > > >> included >> > > > > > >> > > > > battery >> > > > > > >> > > > > > > for >> > > > > > >> > > > > > > > > > > > something >> > > > > > >> > > > > > > > > > > > >> more powerful or tailored to their >> use >> > > case. >> > > > > > The >> > > > > > >> SPI >> > > > > > >> > > > > design >> > > > > > >> > > > > > > > needs >> > > > > > >> > > > > > > > > to >> > > > > > >> > > > > > > > > > > > >> enable >> > > > > > >> > > > > > > > > > > > >> both. >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> The inputs that shaped this framing: >> > > > > > >> > > > > > > > > > > > >> • Anand described how his team >> uses >> > the >> > > > > > current >> > > > > > >> > > > metrics >> > > > > > >> > > > > > > > > > persistence >> > > > > > >> > > > > > > > > > > > >> (three metrics consumers in v1.4). >> > > > > > >> > > > > > > > > > > > >> • Yufei raised Grafana and >> dashboard >> > > > > > >> integrations >> > > > > > >> > > as a >> > > > > > >> > > > > > > > > destination >> > > > > > >> > > > > > > > > > > use >> > > > > > >> > > > > > > > > > > > >> case beyond the default. >> > > > > > >> > > > > > > > > > > > >> • Robert called out that the >> current >> > > design >> > > > > > is >> > > > > > >> > more >> > > > > > >> > > > > > > > > JDBC-focused. >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> Two concrete instances: >> > > > > > >> > > > > > > > > > > > >> • Async metrics intake: Yufei's >> > initial >> > > > > > >> position >> > > > > > >> > was >> > > > > > >> > > > > that >> > > > > > >> > > > > > > > async >> > > > > > >> > > > > > > > > > > should >> > > > > > >> > > > > > > > > > > > >> largely live on the producer side >> and >> > > there >> > > > > is >> > > > > > >> not >> > > > > > >> > > much >> > > > > > >> > > > > > > Polaris >> > > > > > >> > > > > > > > > can >> > > > > > >> > > > > > > > > > > do. >> > > > > > >> > > > > > > > > > > > >> Robert suggested a Polaris-side >> default >> > > is >> > > > > > doable >> > > > > > >> > via >> > > > > > >> > > > > > Vert.x. >> > > > > > >> > > > > > > > > Dmitri >> > > > > > >> > > > > > > > > > > > >> agreed >> > > > > > >> > > > > > > > > > > > >> the direction is worth exploring. >> The >> > > meeting >> > > > > > >> > > converged >> > > > > > >> > > > > on a >> > > > > > >> > > > > > > > > > > > >> battery-included default (likely >> > > > > Vert.x-backed) >> > > > > > >> with >> > > > > > >> > > an >> > > > > > >> > > > > SPI >> > > > > > >> > > > > > > > shape >> > > > > > >> > > > > > > > > > that >> > > > > > >> > > > > > > > > > > > >> lets >> > > > > > >> > > > > > > > > > > > >> power users route to a more scalable >> > > backend >> > > > > > >> > > (k8s-hosted >> > > > > > >> > > > > > > queue, >> > > > > > >> > > > > > > > > AWS >> > > > > > >> > > > > > > > > > > SQS, >> > > > > > >> > > > > > > > > > > > >> etc.). >> > > > > > >> > > > > > > > > > > > >> • Pluggable destinations: >> combining >> > > Yufei's >> > > > > > >> > > dashboard >> > > > > > >> > > > > use >> > > > > > >> > > > > > > case >> > > > > > >> > > > > > > > > > with >> > > > > > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the >> > > meeting >> > > > > > >> agreed >> > > > > > >> > the >> > > > > > >> > > > SPI >> > > > > > >> > > > > > > > should >> > > > > > >> > > > > > > > > be >> > > > > > >> > > > > > > > > > > > >> structured for multiple sinks so >> > > integrations >> > > > > > >> become >> > > > > > >> > > > impl >> > > > > > >> > > > > > > > choices >> > > > > > >> > > > > > > > > > > rather >> > > > > > >> > > > > > > > > > > > >> than architectural changes. >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> The battery-included default is most >> > > likely >> > > > > to >> > > > > > >> use >> > > > > > >> > the >> > > > > > >> > > > > > > existing >> > > > > > >> > > > > > > > > > > > >> JDBC-backed >> > > > > > >> > > > > > > > > > > > >> approach. >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> *Direction (idea-level alignment)* >> > > > > > >> > > > > > > > > > > > >> • Single metrics_report table >> > > consolidating >> > > > > > >> > per-type >> > > > > > >> > > > > > > metrics, >> > > > > > >> > > > > > > > > > > > replacing >> > > > > > >> > > > > > > > > > > > >> scan_metrics_report and >> > > commit_metrics_report >> > > > > > >> > > > > > > > > > > > >> • Iceberg-focused naming and >> schema >> > for >> > > > > now, >> > > > > > >> > revisit >> > > > > > >> > > > if >> > > > > > >> > > > > > > > > > > generic-table >> > > > > > >> > > > > > > > > > > > or >> > > > > > >> > > > > > > > > > > > >> operational metrics arrive >> > > > > > >> > > > > > > > > > > > >> • Metrics persistence as a >> separated >> > > SPI, >> > > > > not >> > > > > > >> on >> > > > > > >> > > > > > > > BasePersistence >> > > > > > >> > > > > > > > > > > > >> • Bootstrap path separated for >> > metrics, >> > > > > > >> > independent >> > > > > > >> > > of >> > > > > > >> > > > > > > > metastore >> > > > > > >> > > > > > > > > > > > >> bootstrap >> > > > > > >> > > > > > > > > > > > >> • "Battery-included plus >> pluggable" >> > as >> > > the >> > > > > > SPI >> > > > > > >> > > design >> > > > > > >> > > > > > > > philosophy >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> *Open items* >> > > > > > >> > > > > > > > > > > > >> • Schema details: >> > > metric_schema_version, >> > > > > > >> payload >> > > > > > >> > > > format, >> > > > > > >> > > > > > IRC >> > > > > > >> > > > > > > > v2 >> > > > > > >> > > > > > > > > > > > >> forward-compat shape >> > > > > > >> > > > > > > > > > > > >> • SPI design details — full review >> > > either >> > > > > in >> > > > > > >> the >> > > > > > >> > > next >> > > > > > >> > > > > sync >> > > > > > >> > > > > > > or >> > > > > > >> > > > > > > > in >> > > > > > >> > > > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> corresponding PR >> > > > > > >> > > > > > > > > > > > >> • Schema refactor PR ownership >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> *Action items* >> > > > > > >> > > > > > > > > > > > >> • EJ to take a first stab at the >> SPI >> > > design >> > > > > > and >> > > > > > >> > > > > > potentially >> > > > > > >> > > > > > > > > > partner >> > > > > > >> > > > > > > > > > > > with >> > > > > > >> > > > > > > > > > > > >> Anand to incorporate the lessons >> > learned >> > > from >> > > > > > the >> > > > > > >> > > > existing >> > > > > > >> > > > > > > > > reporter >> > > > > > >> > > > > > > > > > > and >> > > > > > >> > > > > > > > > > > > >> persistence work. >> > > > > > >> > > > > > > > > > > > >> • Schema refactor PR ownership is >> not >> > > yet >> > > > > > >> decided. >> > > > > > >> > > If >> > > > > > >> > > > > > anyone >> > > > > > >> > > > > > > > is >> > > > > > >> > > > > > > > > > > > >> interested in driving it, reply on >> this >> > > > > thread. >> > > > > > >> > > > > > > > > > > > >> • JB to schedule the next sync, >> > > tentatively >> > > > > > in >> > > > > > >> two >> > > > > > >> > > > > weeks. >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> -ej >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ >> > Wang < >> > > > > > >> > > > > > > > > > > [email protected] >> > > > > > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > >> wrote: >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> > Thanks Yufei for the +1. >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > JB, could you help add a biweekly >> > > metrics >> > > > > > >> > > architecture >> > > > > > >> > > > > > sync >> > > > > > >> > > > > > > to >> > > > > > >> > > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> Polaris >> > > > > > >> > > > > > > > > > > > >> > community calendar? I'm thinking >> > > Thursdays >> > > > > at >> > > > > > >> > 9-10am >> > > > > > >> > > > PT, >> > > > > > >> > > > > > on >> > > > > > >> > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> off-weeks >> > > > > > >> > > > > > > > > > > > >> > from the community meeting >> (starting >> > > May >> > > > > 7), >> > > > > > 60 >> > > > > > >> > > > minutes. >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > Here's a rough agenda to work >> through >> > > over >> > > > > > the >> > > > > > >> > first >> > > > > > >> > > > few >> > > > > > >> > > > > > > > > sessions, >> > > > > > >> > > > > > > > > > > > >> grouped >> > > > > > >> > > > > > > > > > > > >> > by priority: >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > *First: foundational direction* >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > 1. MetricsPersistence: public >> SPI or >> > > > > > internal >> > > > > > >> > > > > > > implementation >> > > > > > >> > > > > > > > > > > detail? >> > > > > > >> > > > > > > > > > > > >> > • Marked @Beta, javadoc calls >> > it a >> > > > > > >> "Service >> > > > > > >> > > > > Provider >> > > > > > >> > > > > > > > > > > Interface", >> > > > > > >> > > > > > > > > > > > >> but >> > > > > > >> > > > > > > > > > > > >> > only one consumer >> > > > > (JdbcBasePersistenceImpl), >> > > > > > >> lives >> > > > > > >> > > on >> > > > > > >> > > > > > > > > > > BasePersistence. >> > > > > > >> > > > > > > > > > > > >> If >> > > > > > >> > > > > > > > > > > > >> > demoted to a private helper >> inside a >> > > > > > persisting >> > > > > > >> > > > reporter >> > > > > > >> > > > > > > impl, >> > > > > > >> > > > > > > > > > most >> > > > > > >> > > > > > > > > > > > >> > downstream design decisions become >> > > > > > >> implementation >> > > > > > >> > > > > details >> > > > > > >> > > > > > > > rather >> > > > > > >> > > > > > > > > > > than >> > > > > > >> > > > > > > > > > > > >> > contract questions. >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > 2. Persistence schema redesign >> > > > > > >> > > > > > > > > > > > >> > • Current two-table layout >> > > > > > >> > > (scan_metrics_report, >> > > > > > >> > > > > > > > > > > > >> > commit_metrics_report) with ~25 >> > > flattened >> > > > > > >> columns >> > > > > > >> > > > each. >> > > > > > >> > > > > > > Every >> > > > > > >> > > > > > > > > new >> > > > > > >> > > > > > > > > > > > metric >> > > > > > >> > > > > > > > > > > > >> > type requires a new table, SPI >> > method, >> > > > > record >> > > > > > >> > class, >> > > > > > >> > > > > > model, >> > > > > > >> > > > > > > > > > > converter, >> > > > > > >> > > > > > > > > > > > >> and >> > > > > > >> > > > > > > > > > > > >> > schema migration. Direction to >> > explore: >> > > > > > single >> > > > > > >> > table >> > > > > > >> > > > > with >> > > > > > >> > > > > > > > > > > metric_type >> > > > > > >> > > > > > > > > > > > >> enum, >> > > > > > >> > > > > > > > > > > > >> > schema_version, and JSON payload >> > > column. >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > *Second: design details once >> > direction >> > > is >> > > > > > set* >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > 3. Partition key strategy >> > > > > > >> > > > > > > > > > > > >> > • Single-table design means >> scan >> > > > > metrics >> > > > > > >> at >> > > > > > >> > > scale >> > > > > > >> > > > > > will >> > > > > > >> > > > > > > > have >> > > > > > >> > > > > > > > > > > high >> > > > > > >> > > > > > > > > > > > >> > write concurrency per table. >> Schema >> > > needs >> > > > > to >> > > > > > >> > expose >> > > > > > >> > > > > enough >> > > > > > >> > > > > > > > > > structure >> > > > > > >> > > > > > > > > > > > for >> > > > > > >> > > > > > > > > > > > >> > backends to shard by entity or >> time >> > > range. >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > 4. Read/write path consistency >> > > > > > >> > > > > > > > > > > > >> > • Writes go through >> > > > > > PolarisMetricsManager >> > > > > > >> on >> > > > > > >> > > > > > > > > > MetaStoreManager. >> > > > > > >> > > > > > > > > > > > >> Reads >> > > > > > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go >> > > straight to >> > > > > > >> > > > > > BasePersistence, >> > > > > > >> > > > > > > > > > > excluding >> > > > > > >> > > > > > > > > > > > >> > non-JDBC backends from the read >> API. >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > *Third: cleanup and alignment* >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > 5. PolarisMetricsReporter naming >> > > > > > >> > > > > > > > > > > > >> > • Only handles IRC >> > > > > > >> (ScanReport/CommitReport), >> > > > > > >> > > > > doesn't >> > > > > > >> > > > > > > > cover >> > > > > > >> > > > > > > > > > > > generic >> > > > > > >> > > > > > > > > > > > >> > tables or operational metrics. >> Name >> > is >> > > > > > broader >> > > > > > >> > than >> > > > > > >> > > > > scope. >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > 6. PolarisMetricsManager facade >> > > > > passthrough >> > > > > > >> > > > > > > > > > > > >> > • Entire default method is >> > > > > > >> > > > > > > > > > > > >> >> > callCtx.getMetaStore().writeScanReport(). >> > > > > > >> > > > > > > > > > > > >> > Zero logic, passes Level 1 >> straight >> > > through >> > > > > > to >> > > > > > >> > Level >> > > > > > >> > > > 3. >> > > > > > >> > > > > > Same >> > > > > > >> > > > > > > > > > > > >> anti-pattern >> > > > > > >> > > > > > > > > > > > >> > as PolarisEventManager. >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > 7. Iceberg community alignment >> > > > > > >> > > > > > > > > > > > >> > • Payload-type extension >> needs >> > > > > > discussion >> > > > > > >> on >> > > > > > >> > > > > > > dev@iceberg. >> > > > > > >> > > > > > > > > > > > >> obelix74's >> > > > > > >> > > > > > > > > > > > >> > Feb thread got zero replies. >> Needs a >> > > > > > committer >> > > > > > >> > > voice. >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > Lets confirm prioritization in the >> > > first >> > > > > > >> session. >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > -ej >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM >> Yufei >> > > Gu < >> > > > > > >> > > > > > > > [email protected]> >> > > > > > >> > > > > > > > > > > > wrote: >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to >> > > drive >> > > > > this >> > > > > > >> > > > forward. I >> > > > > > >> > > > > > > agree >> > > > > > >> > > > > > > > > > that >> > > > > > >> > > > > > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> >> problem is getting complex enough >> > > that a >> > > > > > more >> > > > > > >> > > > > structured >> > > > > > >> > > > > > > > > > discussion >> > > > > > >> > > > > > > > > > > > >> would >> > > > > > >> > > > > > > > > > > > >> >> help. >> > > > > > >> > > > > > > > > > > > >> >> >> > > > > > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync >> for >> > > the >> > > > > > >> metrics >> > > > > > >> > > > > > > > architecture. >> > > > > > >> > > > > > > > > > I’m >> > > > > > >> > > > > > > > > > > > >> happy >> > > > > > >> > > > > > > > > > > > >> >> to >> > > > > > >> > > > > > > > > > > > >> >> join. >> > > > > > >> > > > > > > > > > > > >> >> >> > > > > > >> > > > > > > > > > > > >> >> Yufei >> > > > > > >> > > > > > > > > > > > >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> > > > > > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM >> EJ >> > > Wang < >> > > > > > >> > > > > > > > > > > > >> [email protected]> >> > > > > > >> > > > > > > > > > > > >> >> wrote: >> > > > > > >> > > > > > > > > > > > >> >> >> > > > > > >> > > > > > > > > > > > >> >> > Also, I've been looking more >> > > closely at >> > > > > > the >> > > > > > >> > > > > > *persistence >> > > > > > >> > > > > > > > > schema >> > > > > > >> > > > > > > > > > > in >> > > > > > >> > > > > > > > > > > > >> the >> > > > > > >> > > > > > > > > > > > >> >> > current metrics work*, and I >> think >> > > > > > there's a >> > > > > > >> > > > > structural >> > > > > > >> > > > > > > > > > rigidity >> > > > > > >> > > > > > > > > > > > >> problem >> > > > > > >> > > > > > > > > > > > >> >> > worth raising before the shape >> > gets >> > > > > locked >> > > > > > >> in. >> > > > > > >> > > > > > > > > > > > >> >> > >> > > > > > >> > > > > > > > > > > > >> >> > Right now we have two separate >> > > tables >> > > > > > >> > > > > > > (scan_metrics_report >> > > > > > >> > > > > > > > > and >> > > > > > >> > > > > > > > > > > > >> >> > commit_metrics_report), each >> with >> > > ~25 >> > > > > > >> flattened >> > > > > > >> > > > > columns >> > > > > > >> > > > > > > > that >> > > > > > >> > > > > > > > > > > > directly >> > > > > > >> > > > > > > > > > > > >> >> > mirror the Iceberg report >> fields. >> > > The >> > > > > SPI >> > > > > > >> > follows >> > > > > > >> > > > the >> > > > > > >> > > > > > > same >> > > > > > >> > > > > > > > > > split: >> > > > > > >> > > > > > > > > > > > >> >> > writeScanReport and >> > > writeCommitReport as >> > > > > > >> > separate >> > > > > > >> > > > > > > methods, >> > > > > > >> > > > > > > > > with >> > > > > > >> > > > > > > > > > > > >> per-type >> > > > > > >> > > > > > > > > > > > >> >> > record classes, converters, and >> > > model >> > > > > > >> objects. >> > > > > > >> > > *The >> > > > > > >> > > > > > > > practical >> > > > > > >> > > > > > > > > > > cost: >> > > > > > >> > > > > > > > > > > > >> >> > adding a new metric type >> > > (operational >> > > > > > >> metrics, >> > > > > > >> > > for >> > > > > > >> > > > > > > example) >> > > > > > >> > > > > > > > > > > > requires >> > > > > > >> > > > > > > > > > > > >> a >> > > > > > >> > > > > > > > > > > > >> >> new >> > > > > > >> > > > > > > > > > > > >> >> > table, a new SPI method, a new >> > > record >> > > > > > >> class, a >> > > > > > >> > > new >> > > > > > >> > > > > > model >> > > > > > >> > > > > > > > > > class, a >> > > > > > >> > > > > > > > > > > > new >> > > > > > >> > > > > > > > > > > > >> >> > converter branch, and a schema >> > > > > migration*. >> > > > > > >> > > That's a >> > > > > > >> > > > > lot >> > > > > > >> > > > > > > of >> > > > > > >> > > > > > > > > > > surface >> > > > > > >> > > > > > > > > > > > >> area >> > > > > > >> > > > > > > > > > > > >> >> > for what should be "one more >> kind >> > of >> > > > > > >> metric." >> > > > > > >> > > > > > > > > > > > >> >> > >> > > > > > >> > > > > > > > > > > > >> >> > *My bias* would be toward a >> single >> > > > > metrics >> > > > > > >> > table >> > > > > > >> > > > with >> > > > > > >> > > > > > *a >> > > > > > >> > > > > > > > > typed >> > > > > > >> > > > > > > > > > > JSON >> > > > > > >> > > > > > > > > > > > >> >> > payload*. Something like: >> > > metric_type >> > > > > > >> (enum), >> > > > > > >> > > > > > entity_id, >> > > > > > >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id >> > > > > (nullable), >> > > > > > >> > > > > received_ts, >> > > > > > >> > > > > > > > > > > > >> schema_version, >> > > > > > >> > > > > > > > > > > > >> >> and >> > > > > > >> > > > > > > > > > > > >> >> > a payload column for the >> > > metric-specific >> > > > > > >> data. >> > > > > > >> > > The >> > > > > > >> > > > > > > > > metric_type >> > > > > > >> > > > > > > > > > + >> > > > > > >> > > > > > > > > > > > >> >> > schema_version pair gives us a >> > > > > > >> > forward-compatible >> > > > > > >> > > > > > > contract >> > > > > > >> > > > > > > > > for >> > > > > > >> > > > > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> >> payload >> > > > > > >> > > > > > > > > > > > >> >> > shape. Adding a new metric type >> > > becomes >> > > > > an >> > > > > > >> enum >> > > > > > >> > > > value >> > > > > > >> > > > > > > and a >> > > > > > >> > > > > > > > > > > payload >> > > > > > >> > > > > > > > > > > > >> >> schema, >> > > > > > >> > > > > > > > > > > > >> >> > not a schema migration. One >> thing >> > I >> > > > > think >> > > > > > we >> > > > > > >> > need >> > > > > > >> > > > to >> > > > > > >> > > > > be >> > > > > > >> > > > > > > > > > > deliberate >> > > > > > >> > > > > > > > > > > > >> >> about is >> > > > > > >> > > > > > > > > > > > >> >> > the partition key design. If >> all >> > > metric >> > > > > > >> types >> > > > > > >> > > land >> > > > > > >> > > > in >> > > > > > >> > > > > > one >> > > > > > >> > > > > > > > > > table, >> > > > > > >> > > > > > > > > > > > scan >> > > > > > >> > > > > > > > > > > > >> >> > metrics at scale (high >> > concurrency, >> > > high >> > > > > > >> > > frequency >> > > > > > >> > > > > > across >> > > > > > >> > > > > > > > > many >> > > > > > >> > > > > > > > > > > > >> tables) >> > > > > > >> > > > > > > > > > > > >> >> > could easily create hot >> > partitions. >> > > We'd >> > > > > > >> want >> > > > > > >> > the >> > > > > > >> > > > > > > > persistence >> > > > > > >> > > > > > > > > > > layer >> > > > > > >> > > > > > > > > > > > >> to >> > > > > > >> > > > > > > > > > > > >> >> be >> > > > > > >> > > > > > > > > > > > >> >> > able to shard by entity or time >> > > range, >> > > > > and >> > > > > > >> that >> > > > > > >> > > > means >> > > > > > >> > > > > > the >> > > > > > >> > > > > > > > > > logical >> > > > > > >> > > > > > > > > > > > >> schema >> > > > > > >> > > > > > > > > > > > >> >> > needs to expose enough >> structure >> > for >> > > > > > >> backends >> > > > > > >> > to >> > > > > > >> > > > > > > partition >> > > > > > >> > > > > > > > > on. >> > > > > > >> > > > > > > > > > I >> > > > > > >> > > > > > > > > > > > >> don't >> > > > > > >> > > > > > > > > > > > >> >> > think the current flattened >> layout >> > > gives >> > > > > > us >> > > > > > >> > that. >> > > > > > >> > > > > > > > > > > > >> >> > >> > > > > > >> > > > > > > > > > > > >> >> > This is getting complex enough >> > that >> > > I >> > > > > > don't >> > > > > > >> > think >> > > > > > >> > > > > > ad-hoc >> > > > > > >> > > > > > > > > PR/ML >> > > > > > >> > > > > > > > > > > > >> threads >> > > > > > >> > > > > > > > > > > > >> >> > will converge well. *Would >> people >> > be >> > > > > open >> > > > > > >> to a >> > > > > > >> > > > > biweekly >> > > > > > >> > > > > > > > sync >> > > > > > >> > > > > > > > > > for >> > > > > > >> > > > > > > > > > > > >> metrics >> > > > > > >> > > > > > > > > > > > >> >> > architecture?* I think 30 >> minutes >> > > every >> > > > > > two >> > > > > > >> > weeks >> > > > > > >> > > > > with >> > > > > > >> > > > > > > > > > interested >> > > > > > >> > > > > > > > > > > > >> >> parties >> > > > > > >> > > > > > > > > > > > >> >> > would be enough to work through >> > the >> > > > > > schema, >> > > > > > >> SPI >> > > > > > >> > > > > shape, >> > > > > > >> > > > > > > and >> > > > > > >> > > > > > > > > read >> > > > > > >> > > > > > > > > > > API >> > > > > > >> > > > > > > > > > > > >> >> design >> > > > > > >> > > > > > > > > > > > >> >> > together. Happy to help set >> that >> > up. >> > > > > > >> > > > > > > > > > > > >> >> > >> > > > > > >> > > > > > > > > > > > >> >> > -ej >> > > > > > >> > > > > > > > > > > > >> >> > >> > > > > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at >> 2:19 PM EJ >> > > Wang >> > > > > < >> > > > > > >> > > > > > > > > > > > >> [email protected] >> > > > > > >> > > > > > > > > > > > >> >> > >> > > > > > >> > > > > > > > > > > > >> >> > wrote: >> > > > > > >> > > > > > > > > > > > >> >> > >> > > > > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a >> comment on >> > > the >> > > > > > code >> > > > > > >> > > > > > organization >> > > > > > >> > > > > > > > > side. >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> One thing stood out: the >> metrics >> > > write >> > > > > > path >> > > > > > >> > > enters >> > > > > > >> > > > > > > through >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on >> > > > > > MetaStoreManager, >> > > > > > >> but >> > > > > > >> > > the >> > > > > > >> > > > > new >> > > > > > >> > > > > > > > read >> > > > > > >> > > > > > > > > > path >> > > > > > >> > > > > > > > > > > > >> >> bypasses >> > > > > > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and >> > goes >> > > > > > >> straight to >> > > > > > >> > > > > > > > > BasePersistence >> > > > > > >> > > > > > > > > > > via >> > > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). >> That >> > > means >> > > > > > the >> > > > > > >> > read >> > > > > > >> > > > API >> > > > > > >> > > > > > only >> > > > > > >> > > > > > > > > works >> > > > > > >> > > > > > > > > > > for >> > > > > > >> > > > > > > > > > > > >> >> backends >> > > > > > >> > > > > > > > > > > > >> >> >> that implement >> BasePersistence. >> > > NoSQL >> > > > > and >> > > > > > >> > remote >> > > > > > >> > > > > > > backends >> > > > > > >> > > > > > > > > > can't >> > > > > > >> > > > > > > > > > > > >> >> participate. >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> Stepping back, I think the >> > metrics >> > > > > > >> subsystem >> > > > > > >> > is >> > > > > > >> > > > > > growing >> > > > > > >> > > > > > > > into >> > > > > > >> > > > > > > > > > > > >> something >> > > > > > >> > > > > > > > > > > > >> >> >> real (write + read + REST API >> + >> > > AuthZ + >> > > > > > >> > > > pagination) >> > > > > > >> > > > > > *but >> > > > > > >> > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> >> persistence >> > > > > > >> > > > > > > > > > > > >> >> >> side is split across two >> layers >> > in >> > > a >> > > > > way >> > > > > > >> > that's >> > > > > > >> > > > hard >> > > > > > >> > > > > > to >> > > > > > >> > > > > > > > > > > extend*. I >> > > > > > >> > > > > > > > > > > > >> put >> > > > > > >> > > > > > > > > > > > >> >> >> together two diagrams to show >> > what >> > > I >> > > > > mean >> > > > > > >> (my >> > > > > > >> > > best >> > > > > > >> > > > > > > > effort). >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1): >> > three >> > > > > > >> interfaces >> > > > > > >> > at >> > > > > > >> > > > > three >> > > > > > >> > > > > > > > > > different >> > > > > > >> > > > > > > > > > > > >> >> levels. >> > > > > > >> > > > > > > > > > > > >> >> >> The engine-facing SPI >> > > > > > >> (PolarisMetricsReporter) >> > > > > > >> > > is >> > > > > > >> > > > > > clean. >> > > > > > >> > > > > > > > But >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on >> > > > > MetaStoreManager >> > > > > > >> is a >> > > > > > >> > > > > > > passthrough >> > > > > > >> > > > > > > > > to >> > > > > > >> > > > > > > > > > > > >> >> >> MetricsPersistence on >> > > BasePersistence. >> > > > > > The >> > > > > > >> > @Beta >> > > > > > >> > > > > > > > annotation >> > > > > > >> > > > > > > > > > and >> > > > > > >> > > > > > > > > > > > SPI >> > > > > > >> > > > > > > > > > > > >> >> javadoc >> > > > > > >> > > > > > > > > > > > >> >> >> are on the BasePersistence >> layer, >> > > while >> > > > > > the >> > > > > > >> > > actual >> > > > > > >> > > > > > > > extension >> > > > > > >> > > > > > > > > > > > points >> > > > > > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter, >> > > > > > >> > PolarisMetricsManager) >> > > > > > >> > > > > carry >> > > > > > >> > > > > > no >> > > > > > >> > > > > > > > > > > stability >> > > > > > >> > > > > > > > > > > > >> >> >> annotation. The write path >> goes >> > > through >> > > > > > the >> > > > > > >> > > > > > > > MetaStoreManager >> > > > > > >> > > > > > > > > > > > layer, >> > > > > > >> > > > > > > > > > > > >> the >> > > > > > >> > > > > > > > > > > > >> >> >> read path doesn't. >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2): >> > two >> > > SPIs >> > > > > > at >> > > > > > >> two >> > > > > > >> > > > > levels. >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays >> as >> > the >> > > > > > >> > > engine-facing >> > > > > > >> > > > > SPI. >> > > > > > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes >> the >> > > > > > >> > backend-facing >> > > > > > >> > > > SPI >> > > > > > >> > > > > > > with >> > > > > > >> > > > > > > > > both >> > > > > > >> > > > > > > > > > > > write >> > > > > > >> > > > > > > > > > > > >> >> and >> > > > > > >> > > > > > > > > > > > >> >> >> read methods at the >> > > MetaStoreManager >> > > > > > level, >> > > > > > >> > > where >> > > > > > >> > > > > any >> > > > > > >> > > > > > > > > backend >> > > > > > >> > > > > > > > > > > > (JDBC, >> > > > > > >> > > > > > > > > > > > >> >> NoSQL, >> > > > > > >> > > > > > > > > > > > >> >> >> remote) can implement them. >> > > > > > >> MetricsPersistence >> > > > > > >> > > on >> > > > > > >> > > > > > > > > > > BasePersistence >> > > > > > >> > > > > > > > > > > > >> goes >> > > > > > >> > > > > > > > > > > > >> >> >> away. Where metrics actually >> land >> > > is an >> > > > > > >> > > > > implementation >> > > > > > >> > > > > > > > > detail, >> > > > > > >> > > > > > > > > > > > not a >> > > > > > >> > > > > > > > > > > > >> >> core >> > > > > > >> > > > > > > > > > > > >> >> >> interface. >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> *Minor naming thing*: >> > > > > > >> PolarisMetricsReporter >> > > > > > >> > is >> > > > > > >> > > > > > broader >> > > > > > >> > > > > > > > than >> > > > > > >> > > > > > > > > > > what >> > > > > > >> > > > > > > > > > > > it >> > > > > > >> > > > > > > > > > > > >> >> >> actually handles. It only >> accepts >> > > > > Iceberg >> > > > > > >> REST >> > > > > > >> > > > > Catalog >> > > > > > >> > > > > > > > > metrics >> > > > > > >> > > > > > > > > > > > >> >> (ScanReport, >> > > > > > >> > > > > > > > > > > > >> >> >> CommitReport via >> MetricsReport). >> > > > > Generic >> > > > > > >> table >> > > > > > >> > > > > metrics >> > > > > > >> > > > > > > or >> > > > > > >> > > > > > > > > > > > >> operational >> > > > > > >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not >> > > blocking, >> > > > > > but >> > > > > > >> > worth >> > > > > > >> > > > > > noting >> > > > > > >> > > > > > > if >> > > > > > >> > > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> metrics >> > > > > > >> > > > > > > > > > > > >> >> >> surface expands. >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get >> > there*: >> > > > > > >> > > > > > > > > > > > >> >> >> 1. Add read methods to >> > > > > > >> PolarisMetricsManager >> > > > > > >> > > > > > > > > > (listScanReports, >> > > > > > >> > > > > > > > > > > > >> >> >> listCommitReports) with >> default >> > > no-op, >> > > > > > >> same as >> > > > > > >> > > the >> > > > > > >> > > > > > > > existing >> > > > > > >> > > > > > > > > > > write >> > > > > > >> > > > > > > > > > > > >> >> methods. >> > > > > > >> > > > > > > > > > > > >> >> >> (Probably make >> > > PolarisMetricsManager >> > > > > more >> > > > > > >> > > explicit >> > > > > > >> > > > > on >> > > > > > >> > > > > > > > being >> > > > > > >> > > > > > > > > > > > Iceberg >> > > > > > >> > > > > > > > > > > > >> >> >> specific like package name or >> > class >> > > > > name >> > > > > > >> etc.) >> > > > > > >> > > > > > > > > > > > >> >> >> 2. Wire >> MetricsReportsService >> > > through >> > > > > > >> > > > > > MetaStoreManager >> > > > > > >> > > > > > > > > > instead >> > > > > > >> > > > > > > > > > > > of >> > > > > > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). >> > > > > > >> > > > > > > > > > > > >> >> >> 3. Extract metrics >> persistence >> > > from >> > > > > > >> > > > > > > > > JdbcBasePersistenceImpl >> > > > > > >> > > > > > > > > > > into >> > > > > > >> > > > > > > > > > > > >> its >> > > > > > >> > > > > > > > > > > > >> >> >> own class. That file carries >> ~7 >> > > > > > >> > > responsibilities, >> > > > > > >> > > > > > > metrics >> > > > > > >> > > > > > > > > > being >> > > > > > >> > > > > > > > > > > > one >> > > > > > >> > > > > > > > > > > > >> of >> > > > > > >> > > > > > > > > > > > >> >> them. >> > > > > > >> > > > > > > > > > > > >> >> >> 4. Remove MetricsPersistence >> > from >> > > > > > >> > > > BasePersistence. >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> *None of this needs to happen >> in >> > > #4115. >> > > > > > >> But if >> > > > > > >> > > the >> > > > > > >> > > > > > > > direction >> > > > > > >> > > > > > > > > > > makes >> > > > > > >> > > > > > > > > > > > >> >> sense, >> > > > > > >> > > > > > > > > > > > >> >> >> it would be good to align >> before >> > > the >> > > > > > >> metrics >> > > > > > >> > > > surface >> > > > > > >> > > > > > > grows >> > > > > > >> > > > > > > > > > > > further. >> > > > > > >> > > > > > > > > > > > >> >> Curious >> > > > > > >> > > > > > > > > > > > >> >> >> what others think.* >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> *My mental model note*: Level >> 1 >> > > > > > >> > > MetaStoreManager; >> > > > > > >> > > > > > level >> > > > > > >> > > > > > > 2 >> > > > > > >> > > > > > > > > > > > >> transactional >> > > > > > >> > > > > > > > > > > > >> >> >> persistence; level 3 base >> > > persistence >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 1 >> > > > > > >> > > > > > > > > > > > >> >> >> < >> > > > > > >> > > > > > > > > > > > >> >> >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > >> > >> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXuTBISHg$ >> > > > > > >> > > > > > > > > > > > >> >> > >> > > > > > >> > > > > > > > > > > > >> >> >> : >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png] >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> Diagram 2 >> > > > > > >> > > > > > > > > > > > >> >> >> < >> > > > > > >> > > > > > > > > > > > >> >> >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > >> > >> https://urldefense.com/v3/__https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXSpmqcNk$ >> > > > > > >> > > > > > > > > > > > >> >> > >> > > > > > >> > > > > > > > > > > > >> >> >> : >> > > > > > >> > > > > > > > > > > > >> >> >> [image: image.png] >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> -ej >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at >> 8:22 AM >> > > Dmitri >> > > > > > >> > > > Bourlatchkov >> > > > > > >> > > > > < >> > > > > > >> > > > > > > > > > > > >> [email protected]> >> > > > > > >> > > > > > > > > > > > >> >> >> wrote: >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >>> Hi All, >> > > > > > >> > > > > > > > > > > > >> >> >>> >> > > > > > >> > > > > > > > > > > > >> >> >>> Heads up: The current state >> of >> > PR >> > > > > [4115] >> > > > > > >> > looks >> > > > > > >> > > > > pretty >> > > > > > >> > > > > > > > solid >> > > > > > >> > > > > > > > > > to >> > > > > > >> > > > > > > > > > > > me. >> > > > > > >> > > > > > > > > > > > >> I >> > > > > > >> > > > > > > > > > > > >> >> >>> believe this PR is >> approaching a >> > > > > > mergeable >> > > > > > >> > > > > condition. >> > > > > > >> > > > > > > > > > > > >> >> >>> >> > > > > > >> > > > > > > > > > > > >> >> >>> Please post your reviews if >> you >> > > have >> > > > > any >> > > > > > >> > > > comments. >> > > > > > >> > > > > > > > > > > > >> >> >>> >> > > > > > >> > > > > > > > > > > > >> >> >>> [4115] >> > > > > > >> > > > >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXtFRscKs$ >> > > > > > >> > > > > > > > > > > > >> >> >>> >> > > > > > >> > > > > > > > > > > > >> >> >>> Thanks, >> > > > > > >> > > > > > > > > > > > >> >> >>> Dmitri. >> > > > > > >> > > > > > > > > > > > >> >> >>> >> > > > > > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at >> 3:29 PM >> > > Anand >> > > > > > Kumar >> > > > > > >> > > > Sankaran >> > > > > > >> > > > > > via >> > > > > > >> > > > > > > > > dev < >> > > > > > >> > > > > > > > > > > > >> >> >>> [email protected]> >> wrote: >> > > > > > >> > > > > > > > > > > > >> >> >>> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri, >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the >> > REST >> > > > > > >> endpoints >> > > > > > >> > for >> > > > > > >> > > > > > metrics >> > > > > > >> > > > > > > > and >> > > > > > >> > > > > > > > > > > > events. >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924/changes__;!!Iz9xO38YGHZK!4rnlVIqBQQQ6xpGe1PMV58edOS_cGrKoqdfhw4qSp_JQCavvEjaxXk0OwGPnxyEcO60AzKQK_XDtp7H3DdkFXcrXcjoPhSs$ >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I did not see any >> precursors >> > for >> > > > > > >> raising a >> > > > > > >> > PR >> > > > > > >> > > > for >> > > > > > >> > > > > > > > > > proposals, >> > > > > > >> > > > > > > > > > > so >> > > > > > >> > > > > > > > > > > > >> >> trying >> > > > > > >> > > > > > > > > > > > >> >> >>> > this. Please let me know >> what >> > > you >> > > > > > >> think. >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > - >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran >> < >> > > > > > >> > > > > > > [email protected] >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, >> 2026 at >> > > > > > 10:25 AM >> > > > > > >> > > > > > > > > > > > >> >> >>> > To: [email protected] >> < >> > > > > > >> > > > > [email protected] >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris >> Telemetry >> > > and >> > > > > > Audit >> > > > > > >> > > Trail >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the REST API, based >> on >> > my >> > > use >> > > > > > >> cases: >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > 1. >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query >> > > commit >> > > > > > >> metrics >> > > > > > >> > to >> > > > > > >> > > > > track >> > > > > > >> > > > > > > > files >> > > > > > >> > > > > > > > > > > added >> > > > > > >> > > > > > > > > > > > / >> > > > > > >> > > > > > > > > > > > >> >> >>> removed >> > > > > > >> > > > > > > > > > > > >> >> >>> > per commit, along with >> record >> > > > > counts. >> > > > > > >> The >> > > > > > >> > > > > ingestion >> > > > > > >> > > > > > > > > > pipeline >> > > > > > >> > > > > > > > > > > > that >> > > > > > >> > > > > > > > > > > > >> >> >>> writes >> > > > > > >> > > > > > > > > > > > >> >> >>> > this data is owned by us >> and >> > we >> > > are >> > > > > > >> > > guaranteed >> > > > > > >> > > > to >> > > > > > >> > > > > > > write >> > > > > > >> > > > > > > > > > this >> > > > > > >> > > > > > > > > > > > >> >> >>> information >> > > > > > >> > > > > > > > > > > > >> >> >>> > for each write. >> > > > > > >> > > > > > > > > > > > >> >> >>> > 2. >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query >> > scan >> > > > > > metrics >> > > > > > >> for >> > > > > > >> > > > > read. I >> > > > > > >> > > > > > > > > > > understand >> > > > > > >> > > > > > > > > > > > >> >> clients >> > > > > > >> > > > > > > > > > > > >> >> >>> do >> > > > > > >> > > > > > > > > > > > >> >> >>> > not fulfill this >> requirement. >> > > > > > >> > > > > > > > > > > > >> >> >>> > 3. >> > > > > > >> > > > > > > > > > > > >> >> >>> > I want to be able to query >> the >> > > > > events >> > > > > > >> table >> > > > > > >> > > > > (events >> > > > > > >> > > > > > > are >> > > > > > >> > > > > > > > > > > > >> persisted) - >> > > > > > >> > > > > > > > > > > > >> >> >>> this >> > > > > > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not >> > sure >> > > yet. >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > All this information is in >> the >> > > JDBC >> > > > > > >> based >> > > > > > >> > > > > > persistence >> > > > > > >> > > > > > > > > model >> > > > > > >> > > > > > > > > > > and >> > > > > > >> > > > > > > > > > > > >> is >> > > > > > >> > > > > > > > > > > > >> >> >>> > persisted in the >> metastore. I >> > > > > > currently >> > > > > > >> > don’t >> > > > > > >> > > > > have >> > > > > > >> > > > > > a >> > > > > > >> > > > > > > > need >> > > > > > >> > > > > > > > > > to >> > > > > > >> > > > > > > > > > > > >> query >> > > > > > >> > > > > > > > > > > > >> >> >>> > prometheus or open >> telemetry. >> > I >> > > do >> > > > > > >> publish >> > > > > > >> > > some >> > > > > > >> > > > > > > events >> > > > > > >> > > > > > > > to >> > > > > > >> > > > > > > > > > > > >> Prometheus >> > > > > > >> > > > > > > > > > > > >> >> >>> and >> > > > > > >> > > > > > > > > > > > >> >> >>> > they are forwarded to our >> > > dashboards >> > > > > > >> > > elsewhere. >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I >> > > meant the >> > > > > > >> admin >> > > > > > >> > > user >> > > > > > >> > > > > > > > > utilities. >> > > > > > >> > > > > > > > > > In >> > > > > > >> > > > > > > > > > > > >> one of >> > > > > > >> > > > > > > > > > > > >> >> >>> the >> > > > > > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my >> > proposal, >> > > > > > Prashant >> > > > > > >> > > > > mentioned >> > > > > > >> > > > > > > that >> > > > > > >> > > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> metrics >> > > > > > >> > > > > > > > > > > > >> >> >>> tables >> > > > > > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and >> > that a >> > > > > > similar >> > > > > > >> > > > problem >> > > > > > >> > > > > > > exists >> > > > > > >> > > > > > > > > > with >> > > > > > >> > > > > > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> >> events >> > > > > > >> > > > > > > > > > > > >> >> >>> > table as well. We discussed >> > that >> > > > > > >> cleaning >> > > > > > >> > up >> > > > > > >> > > of >> > > > > > >> > > > > old >> > > > > > >> > > > > > > > > records >> > > > > > >> > > > > > > > > > > > from >> > > > > > >> > > > > > > > > > > > >> >> both >> > > > > > >> > > > > > > > > > > > >> >> >>> > metrics tables and events >> > > tables can >> > > > > > be >> > > > > > >> > done >> > > > > > >> > > > via >> > > > > > >> > > > > a >> > > > > > >> > > > > > > CLI >> > > > > > >> > > > > > > > > > > utility. >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has >> covered >> > the >> > > > > > >> discussion >> > > > > > >> > > > about >> > > > > > >> > > > > > > > > > > datasources. >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > - >> > > > > > >> > > > > > > > > > > > >> >> >>> > Anand >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu < >> > > > > [email protected]> >> > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27, >> > 2026 >> > > at >> > > > > > >> 9:54 PM >> > > > > > >> > > > > > > > > > > > >> >> >>> > To: [email protected] >> < >> > > > > > >> > > > > [email protected] >> > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris >> Telemetry >> > > and >> > > > > > Audit >> > > > > > >> > > Trail >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > This Message Is From an >> > External >> > > > > > Sender >> > > > > > >> > > > > > > > > > > > >> >> >>> > This message came from >> outside >> > > your >> > > > > > >> > > > organization. >> > > > > > >> > > > > > > > > > > > >> >> >>> > Report Suspicious< >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >> > > > > > >> > > > > > > > > > > > >> >> >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > >> > >> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$ >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >> > > > > > >> > > > > > > > > > > > >> >> >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$ >> > >> > > >> > > > > > > > > > > > >> >> >>> , >> > > > > > >> > > > > > > > > > > > >> >> >>> > supporting >> > > > > > >> > > > > > > > > > > > >> >> >>> > multiple data sources is >> not a >> > > > > trivial >> > > > > > >> > > change. >> > > > > > >> > > > I >> > > > > > >> > > > > > > would >> > > > > > >> > > > > > > > > > > strongly >> > > > > > >> > > > > > > > > > > > >> >> >>> recommend >> > > > > > >> > > > > > > > > > > > >> >> >>> > starting with a design >> > document >> > > to >> > > > > > >> > carefully >> > > > > > >> > > > > > evaluate >> > > > > > >> > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> >> architectural >> > > > > > >> > > > > > > > > > > > >> >> >>> > implications and long term >> > > impact. >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query >> > metrics >> > > > > seems >> > > > > > >> > > > reasonable >> > > > > > >> > > > > > > given >> > > > > > >> > > > > > > > > the >> > > > > > >> > > > > > > > > > > > >> current >> > > > > > >> > > > > > > > > > > > >> >> >>> JDBC >> > > > > > >> > > > > > > > > > > > >> >> >>> > based persistence model. >> That >> > > said, >> > > > > we >> > > > > > >> may >> > > > > > >> > > also >> > > > > > >> > > > > > > > consider >> > > > > > >> > > > > > > > > > > > >> alternative >> > > > > > >> > > > > > > > > > > > >> >> >>> > storage models. For >> example, >> > if >> > > we >> > > > > > later >> > > > > > >> > > adopt >> > > > > > >> > > > a >> > > > > > >> > > > > > time >> > > > > > >> > > > > > > > > > series >> > > > > > >> > > > > > > > > > > > >> system >> > > > > > >> > > > > > > > > > > > >> >> >>> such as >> > > > > > >> > > > > > > > > > > > >> >> >>> > Prometheus to store >> metrics, >> > the >> > > > > query >> > > > > > >> > model >> > > > > > >> > > > and >> > > > > > >> > > > > > > access >> > > > > > >> > > > > > > > > > > > patterns >> > > > > > >> > > > > > > > > > > > >> >> would >> > > > > > >> > > > > > > > > > > > >> >> >>> be >> > > > > > >> > > > > > > > > > > > >> >> >>> > fundamentally different. >> > > Designing >> > > > > the >> > > > > > >> REST >> > > > > > >> > > API >> > > > > > >> > > > > > > without >> > > > > > >> > > > > > > > > > > > >> considering >> > > > > > >> > > > > > > > > > > > >> >> >>> these >> > > > > > >> > > > > > > > > > > > >> >> >>> > potential evolutions may >> limit >> > > > > > >> flexibility. >> > > > > > >> > > I'd >> > > > > > >> > > > > > > suggest >> > > > > > >> > > > > > > > > to >> > > > > > >> > > > > > > > > > > > start >> > > > > > >> > > > > > > > > > > > >> >> with >> > > > > > >> > > > > > > > > > > > >> >> >>> the >> > > > > > >> > > > > > > > > > > > >> >> >>> > use case. >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > Yufei >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at >> > 3:42 PM >> > > > > Dmitri >> > > > > > >> > > > > > Bourlatchkov < >> > > > > > >> > > > > > > > > > > > >> >> [email protected]> >> > > > > > >> > > > > > > > > > > > >> >> >>> > wrote: >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand, >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view... >> subject >> > to >> > > > > > >> discussion: >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST >> API >> > to >> > > > > > Polaris >> > > > > > >> is >> > > > > > >> > > > > > perfectly >> > > > > > >> > > > > > > > > fine. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific >> > endpoint >> > > > > URIs >> > > > > > >> and >> > > > > > >> > > > > payloads >> > > > > > >> > > > > > > > might >> > > > > > >> > > > > > > > > > > > require >> > > > > > >> > > > > > > > > > > > >> a >> > > > > > >> > > > > > > > > > > > >> >> few >> > > > > > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a >> > > separate >> > > > > > >> thread >> > > > > > >> > > for >> > > > > > >> > > > > that >> > > > > > >> > > > > > > > might >> > > > > > >> > > > > > > > > > be >> > > > > > >> > > > > > > > > > > > >> best. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly >> create >> > > > > Google >> > > > > > >> Docs >> > > > > > >> > > for >> > > > > > >> > > > > new >> > > > > > >> > > > > > > API >> > > > > > >> > > > > > > > > > > > proposals >> > > > > > >> > > > > > > > > > > > >> too >> > > > > > >> > > > > > > > > > > > >> >> >>> (they >> > > > > > >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as >> the >> > > email >> > > > > > >> > > discussion >> > > > > > >> > > > > > > > > > progresses). >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to >> > try >> > > > > > Markdown >> > > > > > >> > > (with >> > > > > > >> > > > > PRs) >> > > > > > >> > > > > > > for >> > > > > > >> > > > > > > > > > > > proposals >> > > > > > >> > > > > > > > > > > > >> >> [1] >> > > > > > >> > > > > > > > > > > > >> >> >>> ... >> > > > > > >> > > > > > > > > > > > >> >> >>> > > feel free to give it a >> try >> > if >> > > you >> > > > > > are >> > > > > > >> > > > > comfortable >> > > > > > >> > > > > > > > with >> > > > > > >> > > > > > > > > > > that. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify >> whether >> > > you >> > > > > > mean >> > > > > > >> end >> > > > > > >> > > > user >> > > > > > >> > > > > > > > > utilities >> > > > > > >> > > > > > > > > > or >> > > > > > >> > > > > > > > > > > > >> admin >> > > > > > >> > > > > > > > > > > > >> >> >>> user >> > > > > > >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter >> > case >> > > > > those >> > > > > > >> might >> > > > > > >> > > be >> > > > > > >> > > > > more >> > > > > > >> > > > > > > > > > suitable >> > > > > > >> > > > > > > > > > > > for >> > > > > > >> > > > > > > > > > > > >> the >> > > > > > >> > > > > > > > > > > > >> >> >>> Admin >> > > > > > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python >> > CLI, >> > > > > IMHO. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Why would these >> utilities be >> > > > > common >> > > > > > >> with >> > > > > > >> > > > > events? >> > > > > > >> > > > > > > > IMHO, >> > > > > > >> > > > > > > > > > > event >> > > > > > >> > > > > > > > > > > > >> use >> > > > > > >> > > > > > > > > > > > >> >> >>> cases >> > > > > > >> > > > > > > > > > > > >> >> >>> > are >> > > > > > >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit >> > > metrics. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating >> > > metrics >> > > > > > >> > > persistence >> > > > > > >> > > > > from >> > > > > > >> > > > > > > > > > MetaStore >> > > > > > >> > > > > > > > > > > > >> >> >>> persistence >> > > > > > >> > > > > > > > > > > > >> >> >>> > at >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the code level, so that >> they >> > > could >> > > > > > be >> > > > > > >> > mixed >> > > > > > >> > > > and >> > > > > > >> > > > > > > > matched >> > > > > > >> > > > > > > > > > > > >> >> >>> independently. >> > > > > > >> > > > > > > > > > > > >> >> >>> > The >> > > > > > >> > > > > > > > > > > > >> >> >>> > > separate datasource >> question >> > > will >> > > > > > >> become >> > > > > > >> > a >> > > > > > >> > > > > > > non-issue >> > > > > > >> > > > > > > > > with >> > > > > > >> > > > > > > > > > > > that >> > > > > > >> > > > > > > > > > > > >> >> >>> approach, >> > > > > > >> > > > > > > > > > > > >> >> >>> > I >> > > > > > >> > > > > > > > > > > > >> >> >>> > > guess. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for >> separating >> > > scan >> > > > > > >> metrics >> > > > > > >> > > and >> > > > > > >> > > > > > > > metastore >> > > > > > >> > > > > > > > > > > > >> >> persistence >> > > > > > >> > > > > > > > > > > > >> >> >>> is >> > > > > > >> > > > > > > > > > > > >> >> >>> > that >> > > > > > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes" >> between >> > > them >> > > > > are >> > > > > > >> > hardly >> > > > > > >> > > > > ever >> > > > > > >> > > > > > > > > > required. >> > > > > > >> > > > > > > > > > > > >> >> >>> Furthermore, >> > > > > > >> > > > > > > > > > > > >> >> >>> > the >> > > > > > >> > > > > > > > > > > > >> >> >>> > > data and query patterns >> are >> > > very >> > > > > > >> > different >> > > > > > >> > > so >> > > > > > >> > > > > > > > different >> > > > > > >> > > > > > > > > > > > >> >> technologies >> > > > > > >> > > > > > > > > > > > >> >> >>> > might >> > > > > > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each >> case. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [1] >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >> > > > > > >> > > > > > > > > > > > >> >> >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > >> > >> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$ >> > >> > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Cheers, >> > > > > > >> > > > > > > > > > > > >> >> >>> > > Dmitri. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at >> > > 6:19 PM >> > > > > > Anand >> > > > > > >> > Kumar >> > > > > > >> > > > > > > Sankaran >> > > > > > >> > > > > > > > > via >> > > > > > >> > > > > > > > > > > dev >> > > > > > >> > > > > > > > > > > > < >> > > > > > >> > > > > > > > > > > > >> >> >>> > > [email protected]> >> > > wrote: >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is >> > > merged >> > > > > now. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up >> > > features >> > > > > / >> > > > > > >> work >> > > > > > >> > > > > needed. >> > > > > > >> > > > > > > > These >> > > > > > >> > > > > > > > > > > were >> > > > > > >> > > > > > > > > > > > >> all >> > > > > > >> > > > > > > > > > > > >> >> >>> part of >> > > > > > >> > > > > > > > > > > > >> >> >>> > > the >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some >> point in >> > > time >> > > > > > and >> > > > > > >> > were >> > > > > > >> > > > > > removed >> > > > > > >> > > > > > > to >> > > > > > >> > > > > > > > > > > reduce >> > > > > > >> > > > > > > > > > > > >> >> scope. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what >> > you >> > > > > think. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > 1. A REST API to >> > paginate >> > > > > > through >> > > > > > >> > > table >> > > > > > >> > > > > > > metrics. >> > > > > > >> > > > > > > > > > This >> > > > > > >> > > > > > > > > > > > >> will be >> > > > > > >> > > > > > > > > > > > >> >> >>> > non-IRC >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > standard addition. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > 2. Utilities for >> > > managing old >> > > > > > >> > records, >> > > > > > >> > > > > > should >> > > > > > >> > > > > > > be >> > > > > > >> > > > > > > > > > > common >> > > > > > >> > > > > > > > > > > > >> with >> > > > > > >> > > > > > > > > > > > >> >> >>> events. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > There was some >> discussion >> > > that >> > > > > it >> > > > > > >> > belongs >> > > > > > >> > > > to >> > > > > > >> > > > > > the >> > > > > > >> > > > > > > > CLI. >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > 3. Separate >> datasource >> > > > > > (metrics, >> > > > > > >> > > events, >> > > > > > >> > > > > > even >> > > > > > >> > > > > > > > > other >> > > > > > >> > > > > > > > > > > > >> tables?). >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else? >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > - >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > Anand >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > >> > > > > > > > > > > > >> >> >>> >> > > > > > >> > > > > > > > > > > > >> >> >> >> > > > > > >> > > > > > > > > > > > >> >> >> > > > > > >> > > > > > > > > > > > >> > >> > > > > > >> > > > > > > > > > > > >> >> > > > > > >> > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > > >> > > > > > >> > > > > >> > > >> > >> > >> >
