Hi All, It looks like Anand addressed all action items in [4115].
Please review the latest state of the code. The PR LGTM and I approved in GH from my side. [4115] https://github.com/apache/polaris/pull/4115 Thanks, Dmitri. On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > Recap of the discussion in today's Community Sync: > > * The modular approach for Metrics SPI(s) and implementation seems to have > community consensus (please comment if you still have concerns) > > * What modules / feature flags are enabled in Apache Polaris images by > default will be discussed later (when the feature is close to completion). > > In any case, users will be able to turn the full feature on/off at their > discretion, regardless of the default flag values. > > * It is generally preferable to have smaller PRs and introduce new > features gradually to simplify reviews. > > What this means for the Metrics API, IMHO is: > > a) The scope of PR 4115 is probably ok as it stands right now. If someone > wants to propose splitting it further, please comment. > > b) After 4115 is merged, Metrics Persistence PRs will follow, with smaller > scope per PR as appropriate for the nature of the code changes. > > * The major concern was about exposing the new Metrics REST API to end > users. The nature of this concern was about avoiding the (false) impression > that the new API is a firm and adopted Polaris specification. > > I think this concern is reasonable. > > I propose that Anand update the documentation sections of PR 4115 to > emphasise that the API is currently an experimental / POC API at this time > and is subject to change (including breaking changes). I believe it already > has the "beta" label. So this request is mostly for doc changes to clarify > user expectations, since the "beta" label alone may not be interpreted the > same way by all users. > > I hope this update will resolve all remaining concerns with 4115 and allow > for it to be merged so that the community will be able to proceed to > follow-up PRs (as noted above). > > Please add/correct me if I missed anything relevant. > > Thanks, > Dmitri. > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <[email protected]> wrote: > >> I do not think we have agreement on the query REST APIs yet. >> >> In particular, the current query API shape seems highly coupled to the >> *example[1]* metrics in the Iceberg REST spec. I do not think that is a >> good way to design a REST API contract. For example, fields like >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes, >> totalDataManifests, >> totalDeleteManifests, and scannedDataManifests are very specific to one >> style of scan metric. That makes the API look more like a direct >> projection >> of the current example payload than a general query model. >> >> My preference is that Polaris should not define or strongly opine on the >> Iceberg scan or commit metrics query REST API at this point. Polaris can >> resolve and authorize the table, accept the metrics report, and delegate >> it >> to the configured reporting path. Durable storage, filtering, dashboards, >> and query APIs should remain downstream impl. choices unless and until the >> Iceberg community standardizes a query contract, which I doubt will >> happen. >> >> So I do not think the intake path has to be blocked, but I would prefer >> not >> to merge a Polaris specific query API that may be hard to evolve later. >> >> 1. >> >> https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml#L4152 >> >> Yufei >> >> >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <[email protected]> >> wrote: >> >> > Hi All, >> > >> > So, what are the blockers for merging [4115], if any? >> > >> > [4115] https://github.com/apache/polaris/pull/4115 >> > >> > Thanks, >> > Dmitri. >> > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang <[email protected] >> > >> > wrote: >> > >> > > Hi folks, >> > > >> > > Thanks again for the discussion today. I updated the sync doc with >> notes >> > > from today's metrics sync: >> > > >> > > >> > > >> > >> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m >> > > >> > > A few highlights: >> > > >> > > - No objection to keeping the default metrics behavior small: >> > > no-op/log-only is enough as the built-in default. >> > > - Durable storage, event forwarding, external queues, dashboards, >> and >> > > custom filtering should be implementation choices behind the >> metrics >> > > reporting path. >> > > - For PR #4115, the REST layer should resolve/authz/accept the >> report, >> > > then delegate to the selected metrics implementation. It should not >> > own >> > > durable storage semantics. >> > > - The rough boundary we discussed is: spec owns the wire contract, >> > > runtime owns framework wiring, API/contract modules own >> > provider-facing >> > > contracts, and extensions own replaceable implementations. >> > > - OpenLineage has a similar path-naming question. We should avoid >> > > occupying a generic lineage path too early if we want room for >> other >> > > lineage systems later. >> > > >> > > I believe the boundary rules previewed in sync meeting are useful for >> > > reviewing PR #4115, but it is not >> > > a final module-layout proposal. >> > > >> > > For metrics specifically, I think the target remains: >> > > runtime >> > > -> metrics reporting/emitting contract >> > > -> default implementation >> > > -> lower implementation details, if needed >> > > >> > > That keeps the default simple, while allowing durable JDBC, >> event-backed, >> > > or external-queue implementations to be added separately. >> > > >> > > We also touched on the split between Iceberg metrics emitting and >> > > Polaris-owned metrics querying. The querying side can keep evolving, >> > > especially for dashboard or generic-table use cases, but I do not >> think >> > > that needs to block the intake path in this PR. >> > > >> > > Thanks, >> > > -ej >> > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <[email protected]> >> wrote: >> > > >> > > > Thanks for chiming in, EJ. Agreed that the default battery should >> stay >> > > > small. I'm leaning toward not including metrics persistence in the >> > > default >> > > > battery. >> > > > >> > > > Yufei >> > > > >> > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang < >> > [email protected]> >> > > > wrote: >> > > > >> > > > > Hi Yufei, >> > > > > >> > > > > Thanks for connecting this back to the REST endpoint proposal. I >> > > replied >> > > > > with the fuller version on the event-forwarding thread, but >> wanted to >> > > add >> > > > > the shorter version here too. >> > > > > >> > > > > I agree that metrics reporting/emitting is the right conceptual >> > > boundary, >> > > > > and I also think the event/listener path is a good implementation >> > > > direction >> > > > > to explore. The distinction I want to keep clear is default >> battery >> > vs >> > > > > extension implementation. >> > > > > >> > > > > For the REST/API side, I would keep the semantics narrow: Polaris >> > > > resolves >> > > > > and authorizes the table, accepts the Iceberg scan/commit metrics >> > > report >> > > > > into the configured reporting/emitting path, and returns 204. That >> > 204 >> > > > > should mean Polaris accepted the report into the ingestion path; >> it >> > > > should >> > > > > not imply durable storage. >> > > > > >> > > > > Durable storage, event forwarding, external telemetry routing, >> > > filtering, >> > > > > and retention should sit behind that reporting/emitting boundary >> as >> > > > > implementation-layer behavior. A durable JDBC path can be one >> named >> > > > > extension implementation. An event/listener forwarding path can be >> > > > another >> > > > > named extension implementation. >> > > > > >> > > > > So I am +1 on the event-forwarding idea as a non-default extension >> > > > > implementation. I just would not make it the default battery or >> the >> > > core >> > > > > API shape. The default battery can stay small and safe, while >> > > deployments >> > > > > that need forwarding, dashboards, or durable storage can opt into >> the >> > > > > implementation that matches their operational model. >> > > > > >> > > > > Thanks, >> > > > > -ej >> > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <[email protected]> >> > > wrote: >> > > > > >> > > > > > Thanks EJ. I agree that the reporting/emitting boundary feels >> like >> > > the >> > > > > > right SPI boundary, but I wonder if we can simplify this even >> > > further. >> > > > > > >> > > > > > Iceberg scan and commit metrics look very similar to events to >> me. >> > > They >> > > > > are >> > > > > > append only, asynchronously consumed, often forwarded to >> external >> > > > > systems, >> > > > > > and have similar retention and cleanup requirements. More >> details >> > can >> > > > be >> > > > > > found in this thread [1]. In that case, Polaris may only need to >> > > accept >> > > > > the >> > > > > > report and emit an event through the existing event framework. >> > > > Filtering, >> > > > > > async delivery, custom sinks, and retention mechanisms could >> then >> > be >> > > > > shared >> > > > > > instead of introducing a separate metrics specific extension >> layer >> > > and >> > > > > SPI. >> > > > > > >> > > > > > The main requirement I still see is querying metrics through >> > Polaris. >> > > > If >> > > > > we >> > > > > > want that battery included experience, we could work on a >> > persistence >> > > > > > listener implementation that selectively saves metrics and >> expose >> > it >> > > > via >> > > > > > the IRC event endpoint which seems pretty close in Iceberg >> > > > > > community. Another option I like more is to allow users to >> develop >> > > > their >> > > > > > own listener, so they can persist scan/commit metrics to any >> > systems >> > > > they >> > > > > > prefer. In general, that feels like an implementation choice on >> top >> > > of >> > > > > the >> > > > > > event framework rather than something that needs to shape the >> core >> > > > > > architecture. >> > > > > > >> > > > > > 1. >> > https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp >> > > > > > >> > > > > > Thanks, >> > > > > > Yufei >> > > > > > >> > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang < >> > > [email protected] >> > > > > >> > > > > > wrote: >> > > > > > >> > > > > > > Good point. I agree the existing PolarisMetricsReporter is >> > already >> > > > very >> > > > > > > close to the right conceptual boundary. I am not proposing a >> > second >> > > > > > > parallel reporter concept. The distinction I am trying to >> make is >> > > > > > narrower: >> > > > > > > >> > > > > > > Current state: >> > > > > > > >> > > > > > > - PolarisMetricsReporter is the existing metrics reporting >> > hook. >> > > > > > > - But it currently lives under runtime/service. >> > > > > > > - Its contract is documented around Quarkus/CDI discovery >> and >> > > > > > > @Identifier >> > > > > > > selection. >> > > > > > > - The default implementation is log-only. >> > > > > > > - The durable implementation writes through >> > > PolarisMetricsManager >> > > > > and >> > > > > > > MetricsPersistence. >> > > > > > > - MetricsPersistence is currently inherited by >> > BasePersistence. >> > > > > > > >> > > > > > > My proposal: >> > > > > > > >> > > > > > > - Keep the reporting/emitting boundary as the SPI. >> > > > > > > - Revise that contract in a framework-agnostic optional >> > Iceberg >> > > > > > *metrics >> > > > > > > extension API layer*. >> > > > > > > - Keep runtime/service responsible for REST ingestion, >> > > table/authz >> > > > > > > validation, and runtime wiring. >> > > > > > > - Keep the battery implementation log-only in the metrics >> > > > extension >> > > > > > API >> > > > > > > layer, *not under runtime/service*. >> > > > > > > - Treat durable JDBC metrics as one implementation of the >> > > > reporting >> > > > > > SPI. >> > > > > > > - Decompose the MetricsPersistence and BasePersistence >> > coupling >> > > as >> > > > > > > durable implementation detail. >> > > > > > > >> > > > > > > So the difference is not “new reporter concept vs old reporter >> > > > > concept.” >> > > > > > > but: >> > > > > > > >> > > > > > > >> > > > > > > - PolarisMetricsReporter today = runtime/service CDI-shaped >> > > hook. >> > > > > > > - Target reporting SPI = same conceptual boundary, but >> placed >> > in >> > > > the >> > > > > > > right extension API layer and kept framework-agnostic. >> > > > > > > >> > > > > > > For the current metrics PR, I think the minimal architectural >> > > cleanup >> > > > > > > before merge is to *put the reporting SPI contract and the >> > battery >> > > > > > default >> > > > > > > in the right place*. The durable JDBC implementation, >> > event-backed >> > > > > async >> > > > > > > implementation, external queue support, and deeper persistence >> > > > cleanup >> > > > > > can >> > > > > > > continue as follow-ups, as long as we do not lock the SPI >> > boundary >> > > to >> > > > > > > BasePersistence or to runtime/service wiring. >> > > > > > > >> > > > > > > -ej >> > > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov < >> > > [email protected] >> > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Hi EJ, >> > > > > > > > >> > > > > > > > Thanks for the recap / summary! >> > > > > > > > >> > > > > > > > > Do folks agree that the stable SPI boundary should be >> > > > > > > > metrics reporting/emitting, not metrics persistence? >> > > > > > > > >> > > > > > > > SGTM. >> > > > > > > > >> > > > > > > > > Does an optional Iceberg metrics extension API layer sound >> > like >> > > > the >> > > > > > > right >> > > > > > > > home for this SPI? >> > > > > > > > >> > > > > > > > How is that different from current PolarisMetricsReporter? >> > > > > > > > >> > > > > > > > > Should the current durable metrics work be reframed as a >> > > durable >> > > > > > > > JDBC reference implementation of that SPI? >> > > > > > > > >> > > > > > > > SGTM. >> > > > > > > > >> > > > > > > > > What is the smallest PR sequence to get there without >> > blocking >> > > > > > > > the current metrics work unnecessarily? >> > > > > > > > >> > > > > > > > Let's get https://github.com/apache/polaris/pull/4397 >> merged >> > > > first. >> > > > > > > > >> > > > > > > > Then, I'd propose isolating the Metrics schema from the >> > MetaStore >> > > > > > schema. >> > > > > > > > This will probably have some ripple effect into the >> bootstrap >> > > > > > workflows, >> > > > > > > so >> > > > > > > > it's not a trivial change. >> > > > > > > > >> > > > > > > > Then, let's reassess. >> > > > > > > > >> > > > > > > > Cheers, >> > > > > > > > Dmitri. >> > > > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang < >> > > > > [email protected] >> > > > > > > >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > Hi folks, >> > > > > > > > > >> > > > > > > > > Thanks again for the discussion today. I updated the sync >> doc >> > > > with >> > > > > > > notes >> > > > > > > > > from the third metrics architecture sync: >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1 >> > > > > > > > > >> > > > > > > > > A few highlights from today’s discussion: >> > > > > > > > > >> > > > > > > > > - We clarified that Iceberg metrics reporting can be >> > > > interpreted >> > > > > > > > either >> > > > > > > > > as sync or async handling from Polaris’s perspective, >> so >> > > > Polaris >> > > > > > > > should >> > > > > > > > > stay flexible as a platform instead of baking one >> handling >> > > > model >> > > > > > > into >> > > > > > > > > the >> > > > > > > > > REST API behavior. >> > > > > > > > > - We aligned that the built-in (battery) behavior for >> > > metrics >> > > > > > > emitting >> > > > > > > > > can stay simple: no-op/log-only is enough as the >> default. >> > > > > > > > > - Durable metrics persistence should be treated as an >> > > > > > implementation >> > > > > > > > of >> > > > > > > > > the metrics reporting path, not as the core SPI >> boundary. >> > > > > > > > > - The existing durable metrics work can be reviewed as >> a >> > > > > reference >> > > > > > > > > implementation of the reporting SPI, with >> > > persistence-related >> > > > > > logic >> > > > > > > > kept >> > > > > > > > > self-contained under a metrics durable implementation >> > module >> > > > > > rather >> > > > > > > > than >> > > > > > > > > scattered through core entity persistence. >> > > > > > > > > - Dashboard/insights remains a real use case, but we >> > agreed >> > > to >> > > > > > keep >> > > > > > > it >> > > > > > > > > separate from the core metrics intake discussion for >> now. >> > > > > > > > > >> > > > > > > > > I also did a quick source check after the meeting to make >> > sure >> > > we >> > > > > are >> > > > > > > > > describing the current state accurately. >> > > > > > > > > >> > > > > > > > > Current state: >> > > > > > > > > >> > > > > > > > > - Polaris already has a metrics reporting hook: >> > > > > > > > PolarisMetricsReporter. >> > > > > > > > > - The default implementation is DefaultMetricsReporter, >> > > > selected >> > > > > > by >> > > > > > > > > polaris.iceberg-metrics.reporting.type=default. It is >> > > > log-only, >> > > > > > > > > effectively quiet unless metrics logging is enabled. >> > > > > > > > > - There is also a PersistingMetricsReporter, selected >> by >> > > > > > > > > polaris.iceberg-metrics.reporting.type=persisting, >> which >> > > > > converts >> > > > > > > > > Iceberg scan/commit reports into Polaris metrics >> records >> > and >> > > > > > writes >> > > > > > > > > through >> > > > > > > > > PolarisMetricsManager -> MetricsPersistence. >> > > > > > > > > - MetricsPersistence currently exists in the >> persistence >> > > layer >> > > > > and >> > > > > > > is >> > > > > > > > > inherited by BasePersistence. >> > > > > > > > > >> > > > > > > > > My read from the discussion is that the target boundary >> > should >> > > > not >> > > > > be >> > > > > > > > > MetricsPersistence as inherited by BasePersistence. That >> path >> > > > > should >> > > > > > be >> > > > > > > > > decomposed as durable implementation detail (taken care >> of by >> > > > > > > > > https://github.com/apache/polaris/pull/4397). The stable >> > > > extension >> > > > > > > point >> > > > > > > > > should instead be the metrics reporting/emitting boundary. >> > > > > > > > > >> > > > > > > > > Concretely, I think the next proposal should be shaped >> like >> > > this: >> > > > > > > > > >> > > > > > > > > 1. >> > > > > > > > > >> > > > > > > > > Define the proper metrics reporting SPI at an optional >> > > Iceberg >> > > > > > > metrics >> > > > > > > > > extension API layer. >> > > > > > > > > >> > > > > > > > > Example direction: >> > > > > > > > > >> > > > > > > > > public interface IcebergMetricsReporter { >> > > > > > > > > void reportMetric(IcebergMetricsReportContext >> context, >> > > > > > > > > MetricsReport report); >> > > > > > > > > } >> > > > > > > > > >> > > > > > > > > The context would carry the small Polaris-resolved >> > envelope: >> > > > > > > > > catalog/table identity, report type, received >> timestamp, >> > and >> > > > > > > > > request/trace >> > > > > > > > > context if available. The raw Iceberg MetricsReport >> > remains >> > > > the >> > > > > > > > payload. >> > > > > > > > > 2. >> > > > > > > > > >> > > > > > > > > Keep runtime/service as the REST ingestion and wiring >> > layer. >> > > > > > > > > >> > > > > > > > > The REST handler still resolves the table and performs >> > authz >> > > > > > before >> > > > > > > > > accepting the report. After that, it calls the selected >> > > > > reporting >> > > > > > > > > implementation. >> > > > > > > > > 3. >> > > > > > > > > >> > > > > > > > > Keep the battery default no-op/log-only. >> > > > > > > > > >> > > > > > > > > This preserves an out-of-box safe default and avoids >> > > > requiring a >> > > > > > > > durable >> > > > > > > > > metrics store for every Polaris deployment. >> > > > > > > > > 4. >> > > > > > > > > >> > > > > > > > > Move durable JDBC metrics into a self-contained >> > > implementation >> > > > > of >> > > > > > > the >> > > > > > > > > reporting SPI. >> > > > > > > > > >> > > > > > > > > That implementation can own its schema, bootstrap, >> > > retention, >> > > > > and >> > > > > > > read >> > > > > > > > > API support. It should not define the core reporting >> SPI >> > > > > boundary, >> > > > > > > and >> > > > > > > > > it >> > > > > > > > > should not require metrics persistence to remain >> inherited >> > > > from >> > > > > > > > > BasePersistence. >> > > > > > > > > 5. >> > > > > > > > > >> > > > > > > > > Treat async/event-backed handling as another >> > implementation >> > > of >> > > > > the >> > > > > > > > same >> > > > > > > > > reporting SPI. >> > > > > > > > > >> > > > > > > > > For example, an event-backed reporter could enqueue a >> > > metrics >> > > > > > event >> > > > > > > > and >> > > > > > > > > let listeners handle durable storage or other sinks. >> If we >> > > > later >> > > > > > > need >> > > > > > > > a >> > > > > > > > > replaceable queue engine, that seems like a shared >> > > > event/metrics >> > > > > > > > > substrate >> > > > > > > > > topic rather than a metrics-only requirement. >> > > > > > > > > >> > > > > > > > > This framing lets us keep the REST metrics endpoint >> simple, >> > > > > preserve >> > > > > > > the >> > > > > > > > > current default behavior, support durable metrics users, >> and >> > > > still >> > > > > > > leave >> > > > > > > > > room for async/event-backed or external-queue-based >> > > > > implementations. >> > > > > > > > > >> > > > > > > > > I think the main follow-up questions are: >> > > > > > > > > >> > > > > > > > > - Do folks agree that the stable SPI boundary should be >> > > > metrics >> > > > > > > > > reporting/emitting, not metrics persistence? >> > > > > > > > > - Does an optional Iceberg metrics extension API layer >> > sound >> > > > > like >> > > > > > > the >> > > > > > > > > right home for this SPI? >> > > > > > > > > - Should the current durable metrics work be reframed >> as a >> > > > > durable >> > > > > > > > JDBC >> > > > > > > > > reference implementation of that SPI? >> > > > > > > > > - What is the smallest PR sequence to get there without >> > > > blocking >> > > > > > the >> > > > > > > > > current metrics work unnecessarily? >> > > > > > > > > >> > > > > > > > > Thanks, >> > > > > > > > > -ej >> > > > > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov < >> > > > > > [email protected]> >> > > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Hi JB, >> > > > > > > > > > >> > > > > > > > > > Could you set up another meeting, please? Same time on >> > > > Wednesday >> > > > > as >> > > > > > > > last >> > > > > > > > > > time... I hope it works for everyone. >> > > > > > > > > > >> > > > > > > > > > Cheers, >> > > > > > > > > > Dmitri. >> > > > > > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu < >> > > [email protected] >> > > > > >> > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > +1 on another sync call next week. >> > > > > > > > > > > >> > > > > > > > > > > Yufei >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov < >> > > > > > > > [email protected]> >> > > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Hi All, >> > > > > > > > > > > > >> > > > > > > > > > > > WDYT about another sync call next week? >> > > > > > > > > > > > >> > > > > > > > > > > > Thanks, >> > > > > > > > > > > > Dmitri. >> > > > > > > > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri Bourlatchkov < >> > > > > > > > [email protected] >> > > > > > > > > > >> > > > > > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > Hi EJ, >> > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks for the summary! It covers what we >> discussed >> > in >> > > > the >> > > > > > > > meeting >> > > > > > > > > > very >> > > > > > > > > > > > > well, IMHO. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Looking forward to concrete PRs :) >> > > > > > > > > > > > > >> > > > > > > > > > > > > Cheers, >> > > > > > > > > > > > > Dmitri. >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang < >> > > > > > > > > > [email protected] >> > > > > > > > > > > > >> > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > >> Hi folks, >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> We had a community sync earlier, thanks JB for >> > > > scheduling >> > > > > > it. >> > > > > > > > > Notes >> > > > > > > > > > > from >> > > > > > > > > > > > >> the first metrics architecture sync (May 6, >> 10-11am >> > > PT). >> > > > > > > > > Discussion >> > > > > > > > > > > doc >> > > > > > > > > > > > >> with per-section status: >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0 >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> *The meeting covered both topics from the doc. >> > > > > > Direction-level >> > > > > > > > > > > alignment >> > > > > > > > > > > > >> was reached on the headline pieces; details >> remain >> > for >> > > > PR >> > > > > > > review >> > > > > > > > > or >> > > > > > > > > > > > >> follow-up sessions.* >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> *Topic 1 — Persistence schema redesign* >> > > > > > > > > > > > >> Idea-level alignment on consolidating per-type >> > tables >> > > > > > > > > > > > >> (scan_metrics_report, >> > > > > > > > > > > > >> commit_metrics_report) into a single >> metrics_report >> > > > table. >> > > > > > The >> > > > > > > > > > > > motivating >> > > > > > > > > > > > >> cost is the surface area added by every new >> metric >> > > type >> > > > > > today: >> > > > > > > > new >> > > > > > > > > > > > table, >> > > > > > > > > > > > >> SPI method, record class, model, converter, >> schema >> > > > > > migration. >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> Most schema details are deferred to the schema >> PR. A >> > > few >> > > > > > > > specific >> > > > > > > > > > > points >> > > > > > > > > > > > >> came up: >> > > > > > > > > > > > >> • metric_schema_version: Yufei prefers dropping >> > it, >> > > > > since >> > > > > > > > there >> > > > > > > > > is >> > > > > > > > > > > no >> > > > > > > > > > > > >> spec-level concept of metrics versioning today >> and >> > it >> > > is >> > > > > > hard >> > > > > > > to >> > > > > > > > > > > define >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping it, given >> IRC >> > v2 >> > > is >> > > > > > > coming >> > > > > > > > > and >> > > > > > > > > > > the >> > > > > > > > > > > > >> schema should be considered against its likely >> > shape; >> > > > > Robert >> > > > > > > > also >> > > > > > > > > > > raised >> > > > > > > > > > > > >> how to differentiate various payload formats if >> any. >> > > > EJ's >> > > > > > read >> > > > > > > > is >> > > > > > > > > > that >> > > > > > > > > > > > >> this >> > > > > > > > > > > > >> is a two-way-door decision. We can start without >> the >> > > > > field, >> > > > > > > and >> > > > > > > > if >> > > > > > > > > > IRC >> > > > > > > > > > > > v2 >> > > > > > > > > > > > >> changes the shape we would likely roll a >> > corresponding >> > > > new >> > > > > > > > schema >> > > > > > > > > > > > anyway, >> > > > > > > > > > > > >> which is not particularly costly. >> > > > > > > > > > > > >> • Payload format: Robert pointed out that >> future >> > > > formats >> > > > > > > > beyond >> > > > > > > > > > JSON >> > > > > > > > > > > > may >> > > > > > > > > > > > >> be worth supporting. The exact shape is deferred >> to >> > > the >> > > > > > schema >> > > > > > > > > > > > discussion. >> > > > > > > > > > > > >> • Partition strategy: Anand suggested monthly >> > > > > partitioning >> > > > > > > > based >> > > > > > > > > > on >> > > > > > > > > > > > his >> > > > > > > > > > > > >> experience as potentially helpful at scale. >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion and storage >> > belong* >> > > > > > > > > > > > >> Idea-level alignment that metrics should be a >> > > separated >> > > > > SPI >> > > > > > > from >> > > > > > > > > the >> > > > > > > > > > > > >> entity >> > > > > > > > > > > > >> persistence stack. Two reasons surfaced: (a) >> > workloads >> > > > and >> > > > > > > > > > capability >> > > > > > > > > > > > >> requirements diverge enough that coupling them >> > creates >> > > > > > > > artificial >> > > > > > > > > > > > >> constraints, and (b) admin experience improves >> when >> > > > > metrics >> > > > > > > has >> > > > > > > > > its >> > > > > > > > > > > own >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle. Dmitri noted >> > > > Polaris >> > > > > > > being >> > > > > > > > a >> > > > > > > > > > > > platform >> > > > > > > > > > > > >> should have the flexibility to support different >> > > > > persistence >> > > > > > > > > > backends >> > > > > > > > > > > > per >> > > > > > > > > > > > >> concern, and pointed to a concrete next step of >> > > > separating >> > > > > > the >> > > > > > > > > JDBC >> > > > > > > > > > > > >> bootstrap for metrics from the metastore >> bootstrap. >> > > > Robert >> > > > > > > > > proposed >> > > > > > > > > > an >> > > > > > > > > > > > >> additional UX extension: detect an unbootstrapped >> > > > metrics >> > > > > > > store >> > > > > > > > on >> > > > > > > > > > > first >> > > > > > > > > > > > >> use and auto-bootstrap rather than requiring an >> > > explicit >> > > > > > > manual >> > > > > > > > > > > > bootstrap >> > > > > > > > > > > > >> step. >> > > > > > > > > > > > >> The meeting also confirmed that Polaris metrics >> can >> > > > start >> > > > > > > small >> > > > > > > > > and >> > > > > > > > > > > stay >> > > > > > > > > > > > >> Iceberg-focused. Naming and persistence schema >> can >> > > lean >> > > > > > > > > > > > Iceberg-specific. >> > > > > > > > > > > > >> If a future expansion to generic-table metrics or >> > > > > > operational >> > > > > > > > > > metrics >> > > > > > > > > > > > >> arrives, an abstraction layer can be built on >> top of >> > > the >> > > > > > > Iceberg >> > > > > > > > > > > metrics >> > > > > > > > > > > > >> reporter at that point. Robert remains on the >> fence >> > > and >> > > > > > would >> > > > > > > > > prefer >> > > > > > > > > > > > >> something more generic but did not block the >> > > direction; >> > > > > > > Dmitri's >> > > > > > > > > > read >> > > > > > > > > > > > was >> > > > > > > > > > > > >> that the proposed framework already has enough >> > > > flexibility >> > > > > > to >> > > > > > > > > absorb >> > > > > > > > > > > > >> future >> > > > > > > > > > > > >> expansion. >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> The Trade-offs and Proposed structure sections in >> > the >> > > > doc >> > > > > > were >> > > > > > > > not >> > > > > > > > > > > > >> reviewed >> > > > > > > > > > > > >> in detail. They remain open for either the next >> sync >> > > or >> > > > PR >> > > > > > > > review. >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> *Cross-cutting alignment — battery-included plus >> > > > > pluggable* >> > > > > > > > > > > > >> A common philosophy emerged from the discussion. >> EJ >> > > > > > summarized >> > > > > > > > it >> > > > > > > > > > as: >> > > > > > > > > > > > >> Polaris should provide a battery-included UX for >> > > > beginners >> > > > > > and >> > > > > > > > the >> > > > > > > > > > > > >> flexibility for advanced users to swap the >> included >> > > > > battery >> > > > > > > for >> > > > > > > > > > > > something >> > > > > > > > > > > > >> more powerful or tailored to their use case. The >> SPI >> > > > > design >> > > > > > > > needs >> > > > > > > > > to >> > > > > > > > > > > > >> enable >> > > > > > > > > > > > >> both. >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> The inputs that shaped this framing: >> > > > > > > > > > > > >> • Anand described how his team uses the current >> > > > metrics >> > > > > > > > > > persistence >> > > > > > > > > > > > >> (three metrics consumers in v1.4). >> > > > > > > > > > > > >> • Yufei raised Grafana and dashboard >> integrations >> > > as a >> > > > > > > > > destination >> > > > > > > > > > > use >> > > > > > > > > > > > >> case beyond the default. >> > > > > > > > > > > > >> • Robert called out that the current design is >> > more >> > > > > > > > > JDBC-focused. >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> Two concrete instances: >> > > > > > > > > > > > >> • Async metrics intake: Yufei's initial >> position >> > was >> > > > > that >> > > > > > > > async >> > > > > > > > > > > should >> > > > > > > > > > > > >> largely live on the producer side and there is >> not >> > > much >> > > > > > > Polaris >> > > > > > > > > can >> > > > > > > > > > > do. >> > > > > > > > > > > > >> Robert suggested a Polaris-side default is doable >> > via >> > > > > > Vert.x. >> > > > > > > > > Dmitri >> > > > > > > > > > > > >> agreed >> > > > > > > > > > > > >> the direction is worth exploring. The meeting >> > > converged >> > > > > on a >> > > > > > > > > > > > >> battery-included default (likely Vert.x-backed) >> with >> > > an >> > > > > SPI >> > > > > > > > shape >> > > > > > > > > > that >> > > > > > > > > > > > >> lets >> > > > > > > > > > > > >> power users route to a more scalable backend >> > > (k8s-hosted >> > > > > > > queue, >> > > > > > > > > AWS >> > > > > > > > > > > SQS, >> > > > > > > > > > > > >> etc.). >> > > > > > > > > > > > >> • Pluggable destinations: combining Yufei's >> > > dashboard >> > > > > use >> > > > > > > case >> > > > > > > > > > with >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the meeting >> agreed >> > the >> > > > SPI >> > > > > > > > should >> > > > > > > > > be >> > > > > > > > > > > > >> structured for multiple sinks so integrations >> become >> > > > impl >> > > > > > > > choices >> > > > > > > > > > > rather >> > > > > > > > > > > > >> than architectural changes. >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> The battery-included default is most likely to >> use >> > the >> > > > > > > existing >> > > > > > > > > > > > >> JDBC-backed >> > > > > > > > > > > > >> approach. >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> *Direction (idea-level alignment)* >> > > > > > > > > > > > >> • Single metrics_report table consolidating >> > per-type >> > > > > > > metrics, >> > > > > > > > > > > > replacing >> > > > > > > > > > > > >> scan_metrics_report and commit_metrics_report >> > > > > > > > > > > > >> • Iceberg-focused naming and schema for now, >> > revisit >> > > > if >> > > > > > > > > > > generic-table >> > > > > > > > > > > > or >> > > > > > > > > > > > >> operational metrics arrive >> > > > > > > > > > > > >> • Metrics persistence as a separated SPI, not >> on >> > > > > > > > BasePersistence >> > > > > > > > > > > > >> • Bootstrap path separated for metrics, >> > independent >> > > of >> > > > > > > > metastore >> > > > > > > > > > > > >> bootstrap >> > > > > > > > > > > > >> • "Battery-included plus pluggable" as the SPI >> > > design >> > > > > > > > philosophy >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> *Open items* >> > > > > > > > > > > > >> • Schema details: metric_schema_version, >> payload >> > > > format, >> > > > > > IRC >> > > > > > > > v2 >> > > > > > > > > > > > >> forward-compat shape >> > > > > > > > > > > > >> • SPI design details — full review either in >> the >> > > next >> > > > > sync >> > > > > > > or >> > > > > > > > in >> > > > > > > > > > the >> > > > > > > > > > > > >> corresponding PR >> > > > > > > > > > > > >> • Schema refactor PR ownership >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> *Action items* >> > > > > > > > > > > > >> • EJ to take a first stab at the SPI design and >> > > > > > potentially >> > > > > > > > > > partner >> > > > > > > > > > > > with >> > > > > > > > > > > > >> Anand to incorporate the lessons learned from the >> > > > existing >> > > > > > > > > reporter >> > > > > > > > > > > and >> > > > > > > > > > > > >> persistence work. >> > > > > > > > > > > > >> • Schema refactor PR ownership is not yet >> decided. >> > > If >> > > > > > anyone >> > > > > > > > is >> > > > > > > > > > > > >> interested in driving it, reply on this thread. >> > > > > > > > > > > > >> • JB to schedule the next sync, tentatively in >> two >> > > > > weeks. >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> -ej >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang < >> > > > > > > > > > > [email protected] >> > > > > > > > > > > > > >> > > > > > > > > > > > >> wrote: >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> > Thanks Yufei for the +1. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > JB, could you help add a biweekly metrics >> > > architecture >> > > > > > sync >> > > > > > > to >> > > > > > > > > the >> > > > > > > > > > > > >> Polaris >> > > > > > > > > > > > >> > community calendar? I'm thinking Thursdays at >> > 9-10am >> > > > PT, >> > > > > > on >> > > > > > > > the >> > > > > > > > > > > > >> off-weeks >> > > > > > > > > > > > >> > from the community meeting (starting May 7), 60 >> > > > minutes. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > Here's a rough agenda to work through over the >> > first >> > > > few >> > > > > > > > > sessions, >> > > > > > > > > > > > >> grouped >> > > > > > > > > > > > >> > by priority: >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > *First: foundational direction* >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > 1. MetricsPersistence: public SPI or internal >> > > > > > > implementation >> > > > > > > > > > > detail? >> > > > > > > > > > > > >> > • Marked @Beta, javadoc calls it a >> "Service >> > > > > Provider >> > > > > > > > > > > Interface", >> > > > > > > > > > > > >> but >> > > > > > > > > > > > >> > only one consumer (JdbcBasePersistenceImpl), >> lives >> > > on >> > > > > > > > > > > BasePersistence. >> > > > > > > > > > > > >> If >> > > > > > > > > > > > >> > demoted to a private helper inside a persisting >> > > > reporter >> > > > > > > impl, >> > > > > > > > > > most >> > > > > > > > > > > > >> > downstream design decisions become >> implementation >> > > > > details >> > > > > > > > rather >> > > > > > > > > > > than >> > > > > > > > > > > > >> > contract questions. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > 2. Persistence schema redesign >> > > > > > > > > > > > >> > • Current two-table layout >> > > (scan_metrics_report, >> > > > > > > > > > > > >> > commit_metrics_report) with ~25 flattened >> columns >> > > > each. >> > > > > > > Every >> > > > > > > > > new >> > > > > > > > > > > > metric >> > > > > > > > > > > > >> > type requires a new table, SPI method, record >> > class, >> > > > > > model, >> > > > > > > > > > > converter, >> > > > > > > > > > > > >> and >> > > > > > > > > > > > >> > schema migration. Direction to explore: single >> > table >> > > > > with >> > > > > > > > > > > metric_type >> > > > > > > > > > > > >> enum, >> > > > > > > > > > > > >> > schema_version, and JSON payload column. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > *Second: design details once direction is set* >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > 3. Partition key strategy >> > > > > > > > > > > > >> > • Single-table design means scan metrics >> at >> > > scale >> > > > > > will >> > > > > > > > have >> > > > > > > > > > > high >> > > > > > > > > > > > >> > write concurrency per table. Schema needs to >> > expose >> > > > > enough >> > > > > > > > > > structure >> > > > > > > > > > > > for >> > > > > > > > > > > > >> > backends to shard by entity or time range. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > 4. Read/write path consistency >> > > > > > > > > > > > >> > • Writes go through PolarisMetricsManager >> on >> > > > > > > > > > MetaStoreManager. >> > > > > > > > > > > > >> Reads >> > > > > > > > > > > > >> > bypass MetaStoreManager and go straight to >> > > > > > BasePersistence, >> > > > > > > > > > > excluding >> > > > > > > > > > > > >> > non-JDBC backends from the read API. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > *Third: cleanup and alignment* >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > 5. PolarisMetricsReporter naming >> > > > > > > > > > > > >> > • Only handles IRC >> (ScanReport/CommitReport), >> > > > > doesn't >> > > > > > > > cover >> > > > > > > > > > > > generic >> > > > > > > > > > > > >> > tables or operational metrics. Name is broader >> > than >> > > > > scope. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > 6. PolarisMetricsManager facade passthrough >> > > > > > > > > > > > >> > • Entire default method is >> > > > > > > > > > > > >> callCtx.getMetaStore().writeScanReport(). >> > > > > > > > > > > > >> > Zero logic, passes Level 1 straight through to >> > Level >> > > > 3. >> > > > > > Same >> > > > > > > > > > > > >> anti-pattern >> > > > > > > > > > > > >> > as PolarisEventManager. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > 7. Iceberg community alignment >> > > > > > > > > > > > >> > • Payload-type extension needs discussion >> on >> > > > > > > dev@iceberg. >> > > > > > > > > > > > >> obelix74's >> > > > > > > > > > > > >> > Feb thread got zero replies. Needs a committer >> > > voice. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > Lets confirm prioritization in the first >> session. >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > -ej >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu < >> > > > > > > > [email protected]> >> > > > > > > > > > > > wrote: >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to drive this >> > > > forward. I >> > > > > > > agree >> > > > > > > > > > that >> > > > > > > > > > > > the >> > > > > > > > > > > > >> >> problem is getting complex enough that a more >> > > > > structured >> > > > > > > > > > discussion >> > > > > > > > > > > > >> would >> > > > > > > > > > > > >> >> help. >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync for the >> metrics >> > > > > > > > architecture. >> > > > > > > > > > I’m >> > > > > > > > > > > > >> happy >> > > > > > > > > > > > >> >> to >> > > > > > > > > > > > >> >> join. >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> >> Yufei >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang < >> > > > > > > > > > > > >> [email protected]> >> > > > > > > > > > > > >> >> wrote: >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> >> > Also, I've been looking more closely at the >> > > > > > *persistence >> > > > > > > > > schema >> > > > > > > > > > > in >> > > > > > > > > > > > >> the >> > > > > > > > > > > > >> >> > current metrics work*, and I think there's a >> > > > > structural >> > > > > > > > > > rigidity >> > > > > > > > > > > > >> problem >> > > > > > > > > > > > >> >> > worth raising before the shape gets locked >> in. >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> > Right now we have two separate tables >> > > > > > > (scan_metrics_report >> > > > > > > > > and >> > > > > > > > > > > > >> >> > commit_metrics_report), each with ~25 >> flattened >> > > > > columns >> > > > > > > > that >> > > > > > > > > > > > directly >> > > > > > > > > > > > >> >> > mirror the Iceberg report fields. The SPI >> > follows >> > > > the >> > > > > > > same >> > > > > > > > > > split: >> > > > > > > > > > > > >> >> > writeScanReport and writeCommitReport as >> > separate >> > > > > > > methods, >> > > > > > > > > with >> > > > > > > > > > > > >> per-type >> > > > > > > > > > > > >> >> > record classes, converters, and model >> objects. >> > > *The >> > > > > > > > practical >> > > > > > > > > > > cost: >> > > > > > > > > > > > >> >> > adding a new metric type (operational >> metrics, >> > > for >> > > > > > > example) >> > > > > > > > > > > > requires >> > > > > > > > > > > > >> a >> > > > > > > > > > > > >> >> new >> > > > > > > > > > > > >> >> > table, a new SPI method, a new record >> class, a >> > > new >> > > > > > model >> > > > > > > > > > class, a >> > > > > > > > > > > > new >> > > > > > > > > > > > >> >> > converter branch, and a schema migration*. >> > > That's a >> > > > > lot >> > > > > > > of >> > > > > > > > > > > surface >> > > > > > > > > > > > >> area >> > > > > > > > > > > > >> >> > for what should be "one more kind of >> metric." >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> > *My bias* would be toward a single metrics >> > table >> > > > with >> > > > > > *a >> > > > > > > > > typed >> > > > > > > > > > > JSON >> > > > > > > > > > > > >> >> > payload*. Something like: metric_type >> (enum), >> > > > > > entity_id, >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id (nullable), >> > > > > received_ts, >> > > > > > > > > > > > >> schema_version, >> > > > > > > > > > > > >> >> and >> > > > > > > > > > > > >> >> > a payload column for the metric-specific >> data. >> > > The >> > > > > > > > > metric_type >> > > > > > > > > > + >> > > > > > > > > > > > >> >> > schema_version pair gives us a >> > forward-compatible >> > > > > > > contract >> > > > > > > > > for >> > > > > > > > > > > the >> > > > > > > > > > > > >> >> payload >> > > > > > > > > > > > >> >> > shape. Adding a new metric type becomes an >> enum >> > > > value >> > > > > > > and a >> > > > > > > > > > > payload >> > > > > > > > > > > > >> >> schema, >> > > > > > > > > > > > >> >> > not a schema migration. One thing I think we >> > need >> > > > to >> > > > > be >> > > > > > > > > > > deliberate >> > > > > > > > > > > > >> >> about is >> > > > > > > > > > > > >> >> > the partition key design. If all metric >> types >> > > land >> > > > in >> > > > > > one >> > > > > > > > > > table, >> > > > > > > > > > > > scan >> > > > > > > > > > > > >> >> > metrics at scale (high concurrency, high >> > > frequency >> > > > > > across >> > > > > > > > > many >> > > > > > > > > > > > >> tables) >> > > > > > > > > > > > >> >> > could easily create hot partitions. We'd >> want >> > the >> > > > > > > > persistence >> > > > > > > > > > > layer >> > > > > > > > > > > > >> to >> > > > > > > > > > > > >> >> be >> > > > > > > > > > > > >> >> > able to shard by entity or time range, and >> that >> > > > means >> > > > > > the >> > > > > > > > > > logical >> > > > > > > > > > > > >> schema >> > > > > > > > > > > > >> >> > needs to expose enough structure for >> backends >> > to >> > > > > > > partition >> > > > > > > > > on. >> > > > > > > > > > I >> > > > > > > > > > > > >> don't >> > > > > > > > > > > > >> >> > think the current flattened layout gives us >> > that. >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> > This is getting complex enough that I don't >> > think >> > > > > > ad-hoc >> > > > > > > > > PR/ML >> > > > > > > > > > > > >> threads >> > > > > > > > > > > > >> >> > will converge well. *Would people be open >> to a >> > > > > biweekly >> > > > > > > > sync >> > > > > > > > > > for >> > > > > > > > > > > > >> metrics >> > > > > > > > > > > > >> >> > architecture?* I think 30 minutes every two >> > weeks >> > > > > with >> > > > > > > > > > interested >> > > > > > > > > > > > >> >> parties >> > > > > > > > > > > > >> >> > would be enough to work through the schema, >> SPI >> > > > > shape, >> > > > > > > and >> > > > > > > > > read >> > > > > > > > > > > API >> > > > > > > > > > > > >> >> design >> > > > > > > > > > > > >> >> > together. Happy to help set that up. >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> > -ej >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang < >> > > > > > > > > > > > >> [email protected] >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> > wrote: >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a comment on the code >> > > > > > organization >> > > > > > > > > side. >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> One thing stood out: the metrics write path >> > > enters >> > > > > > > through >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager, >> but >> > > the >> > > > > new >> > > > > > > > read >> > > > > > > > > > path >> > > > > > > > > > > > >> >> bypasses >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and goes >> straight to >> > > > > > > > > BasePersistence >> > > > > > > > > > > via >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). That means the >> > read >> > > > API >> > > > > > only >> > > > > > > > > works >> > > > > > > > > > > for >> > > > > > > > > > > > >> >> backends >> > > > > > > > > > > > >> >> >> that implement BasePersistence. NoSQL and >> > remote >> > > > > > > backends >> > > > > > > > > > can't >> > > > > > > > > > > > >> >> participate. >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> Stepping back, I think the metrics >> subsystem >> > is >> > > > > > growing >> > > > > > > > into >> > > > > > > > > > > > >> something >> > > > > > > > > > > > >> >> >> real (write + read + REST API + AuthZ + >> > > > pagination) >> > > > > > *but >> > > > > > > > the >> > > > > > > > > > > > >> >> persistence >> > > > > > > > > > > > >> >> >> side is split across two layers in a way >> > that's >> > > > hard >> > > > > > to >> > > > > > > > > > > extend*. I >> > > > > > > > > > > > >> put >> > > > > > > > > > > > >> >> >> together two diagrams to show what I mean >> (my >> > > best >> > > > > > > > effort). >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1): three >> interfaces >> > at >> > > > > three >> > > > > > > > > > different >> > > > > > > > > > > > >> >> levels. >> > > > > > > > > > > > >> >> >> The engine-facing SPI >> (PolarisMetricsReporter) >> > > is >> > > > > > clean. >> > > > > > > > But >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager >> is a >> > > > > > > passthrough >> > > > > > > > > to >> > > > > > > > > > > > >> >> >> MetricsPersistence on BasePersistence. The >> > @Beta >> > > > > > > > annotation >> > > > > > > > > > and >> > > > > > > > > > > > SPI >> > > > > > > > > > > > >> >> javadoc >> > > > > > > > > > > > >> >> >> are on the BasePersistence layer, while the >> > > actual >> > > > > > > > extension >> > > > > > > > > > > > points >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter, >> > PolarisMetricsManager) >> > > > > carry >> > > > > > no >> > > > > > > > > > > stability >> > > > > > > > > > > > >> >> >> annotation. The write path goes through the >> > > > > > > > MetaStoreManager >> > > > > > > > > > > > layer, >> > > > > > > > > > > > >> the >> > > > > > > > > > > > >> >> >> read path doesn't. >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs at >> two >> > > > > levels. >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as the >> > > engine-facing >> > > > > SPI. >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes the >> > backend-facing >> > > > SPI >> > > > > > > with >> > > > > > > > > both >> > > > > > > > > > > > write >> > > > > > > > > > > > >> >> and >> > > > > > > > > > > > >> >> >> read methods at the MetaStoreManager level, >> > > where >> > > > > any >> > > > > > > > > backend >> > > > > > > > > > > > (JDBC, >> > > > > > > > > > > > >> >> NoSQL, >> > > > > > > > > > > > >> >> >> remote) can implement them. >> MetricsPersistence >> > > on >> > > > > > > > > > > BasePersistence >> > > > > > > > > > > > >> goes >> > > > > > > > > > > > >> >> >> away. Where metrics actually land is an >> > > > > implementation >> > > > > > > > > detail, >> > > > > > > > > > > > not a >> > > > > > > > > > > > >> >> core >> > > > > > > > > > > > >> >> >> interface. >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> *Minor naming thing*: >> PolarisMetricsReporter >> > is >> > > > > > broader >> > > > > > > > than >> > > > > > > > > > > what >> > > > > > > > > > > > it >> > > > > > > > > > > > >> >> >> actually handles. It only accepts Iceberg >> REST >> > > > > Catalog >> > > > > > > > > metrics >> > > > > > > > > > > > >> >> (ScanReport, >> > > > > > > > > > > > >> >> >> CommitReport via MetricsReport). Generic >> table >> > > > > metrics >> > > > > > > or >> > > > > > > > > > > > >> operational >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not blocking, but >> > worth >> > > > > > noting >> > > > > > > if >> > > > > > > > > the >> > > > > > > > > > > > >> metrics >> > > > > > > > > > > > >> >> >> surface expands. >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get there*: >> > > > > > > > > > > > >> >> >> 1. Add read methods to >> PolarisMetricsManager >> > > > > > > > > > (listScanReports, >> > > > > > > > > > > > >> >> >> listCommitReports) with default no-op, >> same as >> > > the >> > > > > > > > existing >> > > > > > > > > > > write >> > > > > > > > > > > > >> >> methods. >> > > > > > > > > > > > >> >> >> (Probably make PolarisMetricsManager more >> > > explicit >> > > > > on >> > > > > > > > being >> > > > > > > > > > > > Iceberg >> > > > > > > > > > > > >> >> >> specific like package name or class name >> etc.) >> > > > > > > > > > > > >> >> >> 2. Wire MetricsReportsService through >> > > > > > MetaStoreManager >> > > > > > > > > > instead >> > > > > > > > > > > > of >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). >> > > > > > > > > > > > >> >> >> 3. Extract metrics persistence from >> > > > > > > > > JdbcBasePersistenceImpl >> > > > > > > > > > > into >> > > > > > > > > > > > >> its >> > > > > > > > > > > > >> >> >> own class. That file carries ~7 >> > > responsibilities, >> > > > > > > metrics >> > > > > > > > > > being >> > > > > > > > > > > > one >> > > > > > > > > > > > >> of >> > > > > > > > > > > > >> >> them. >> > > > > > > > > > > > >> >> >> 4. Remove MetricsPersistence from >> > > > BasePersistence. >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> *None of this needs to happen in #4115. >> But if >> > > the >> > > > > > > > direction >> > > > > > > > > > > makes >> > > > > > > > > > > > >> >> sense, >> > > > > > > > > > > > >> >> >> it would be good to align before the >> metrics >> > > > surface >> > > > > > > grows >> > > > > > > > > > > > further. >> > > > > > > > > > > > >> >> Curious >> > > > > > > > > > > > >> >> >> what others think.* >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> *My mental model note*: Level 1 >> > > MetaStoreManager; >> > > > > > level >> > > > > > > 2 >> > > > > > > > > > > > >> transactional >> > > > > > > > > > > > >> >> >> persistence; level 3 base persistence >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> Diagram 1 >> > > > > > > > > > > > >> >> >> < >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00 >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> >> : >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> [image: image.png] >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> Diagram 2 >> > > > > > > > > > > > >> >> >> < >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> >> : >> > > > > > > > > > > > >> >> >> [image: image.png] >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> -ej >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri >> > > > Bourlatchkov >> > > > > < >> > > > > > > > > > > > >> [email protected]> >> > > > > > > > > > > > >> >> >> wrote: >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >>> Hi All, >> > > > > > > > > > > > >> >> >>> >> > > > > > > > > > > > >> >> >>> Heads up: The current state of PR [4115] >> > looks >> > > > > pretty >> > > > > > > > solid >> > > > > > > > > > to >> > > > > > > > > > > > me. >> > > > > > > > > > > > >> I >> > > > > > > > > > > > >> >> >>> believe this PR is approaching a mergeable >> > > > > condition. >> > > > > > > > > > > > >> >> >>> >> > > > > > > > > > > > >> >> >>> Please post your reviews if you have any >> > > > comments. >> > > > > > > > > > > > >> >> >>> >> > > > > > > > > > > > >> >> >>> [4115] >> > > > https://github.com/apache/polaris/pull/4115 >> > > > > > > > > > > > >> >> >>> >> > > > > > > > > > > > >> >> >>> Thanks, >> > > > > > > > > > > > >> >> >>> Dmitri. >> > > > > > > > > > > > >> >> >>> >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand Kumar >> > > > Sankaran >> > > > > > via >> > > > > > > > > dev < >> > > > > > > > > > > > >> >> >>> [email protected]> wrote: >> > > > > > > > > > > > >> >> >>> >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri, >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the REST >> endpoints >> > for >> > > > > > metrics >> > > > > > > > and >> > > > > > > > > > > > events. >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > >> > > > > > https://github.com/apache/polaris/pull/3924/changes >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > I did not see any precursors for >> raising a >> > PR >> > > > for >> > > > > > > > > > proposals, >> > > > > > > > > > > so >> > > > > > > > > > > > >> >> trying >> > > > > > > > > > > > >> >> >>> > this. Please let me know what you >> think. >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > - >> > > > > > > > > > > > >> >> >>> > Anand >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran < >> > > > > > > [email protected] >> > > > > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at 10:25 AM >> > > > > > > > > > > > >> >> >>> > To: [email protected] < >> > > > > [email protected] >> > > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit >> > > Trail >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > About the REST API, based on my use >> cases: >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > 1. >> > > > > > > > > > > > >> >> >>> > I want to be able to query commit >> metrics >> > to >> > > > > track >> > > > > > > > files >> > > > > > > > > > > added >> > > > > > > > > > > > / >> > > > > > > > > > > > >> >> >>> removed >> > > > > > > > > > > > >> >> >>> > per commit, along with record counts. >> The >> > > > > ingestion >> > > > > > > > > > pipeline >> > > > > > > > > > > > that >> > > > > > > > > > > > >> >> >>> writes >> > > > > > > > > > > > >> >> >>> > this data is owned by us and we are >> > > guaranteed >> > > > to >> > > > > > > write >> > > > > > > > > > this >> > > > > > > > > > > > >> >> >>> information >> > > > > > > > > > > > >> >> >>> > for each write. >> > > > > > > > > > > > >> >> >>> > 2. >> > > > > > > > > > > > >> >> >>> > I want to be able to query scan metrics >> for >> > > > > read. I >> > > > > > > > > > > understand >> > > > > > > > > > > > >> >> clients >> > > > > > > > > > > > >> >> >>> do >> > > > > > > > > > > > >> >> >>> > not fulfill this requirement. >> > > > > > > > > > > > >> >> >>> > 3. >> > > > > > > > > > > > >> >> >>> > I want to be able to query the events >> table >> > > > > (events >> > > > > > > are >> > > > > > > > > > > > >> persisted) - >> > > > > > > > > > > > >> >> >>> this >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not sure yet. >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > All this information is in the JDBC >> based >> > > > > > persistence >> > > > > > > > > model >> > > > > > > > > > > and >> > > > > > > > > > > > >> is >> > > > > > > > > > > > >> >> >>> > persisted in the metastore. I currently >> > don’t >> > > > > have >> > > > > > a >> > > > > > > > need >> > > > > > > > > > to >> > > > > > > > > > > > >> query >> > > > > > > > > > > > >> >> >>> > prometheus or open telemetry. I do >> publish >> > > some >> > > > > > > events >> > > > > > > > to >> > > > > > > > > > > > >> Prometheus >> > > > > > > > > > > > >> >> >>> and >> > > > > > > > > > > > >> >> >>> > they are forwarded to our dashboards >> > > elsewhere. >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I meant the >> admin >> > > user >> > > > > > > > > utilities. >> > > > > > > > > > In >> > > > > > > > > > > > >> one of >> > > > > > > > > > > > >> >> >>> the >> > > > > > > > > > > > >> >> >>> > earliest drafts of my proposal, Prashant >> > > > > mentioned >> > > > > > > that >> > > > > > > > > the >> > > > > > > > > > > > >> metrics >> > > > > > > > > > > > >> >> >>> tables >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and that a similar >> > > > problem >> > > > > > > exists >> > > > > > > > > > with >> > > > > > > > > > > > the >> > > > > > > > > > > > >> >> events >> > > > > > > > > > > > >> >> >>> > table as well. We discussed that >> cleaning >> > up >> > > of >> > > > > old >> > > > > > > > > records >> > > > > > > > > > > > from >> > > > > > > > > > > > >> >> both >> > > > > > > > > > > > >> >> >>> > metrics tables and events tables can be >> > done >> > > > via >> > > > > a >> > > > > > > CLI >> > > > > > > > > > > utility. >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has covered the >> discussion >> > > > about >> > > > > > > > > > > datasources. >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > - >> > > > > > > > > > > > >> >> >>> > Anand >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <[email protected]> >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at >> 9:54 PM >> > > > > > > > > > > > >> >> >>> > To: [email protected] < >> > > > > [email protected] >> > > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and Audit >> > > Trail >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > This Message Is From an External Sender >> > > > > > > > > > > > >> >> >>> > This message came from outside your >> > > > organization. >> > > > > > > > > > > > >> >> >>> > Report Suspicious< >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$ >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > As I mentioned in >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$ >> > > > > > > > > > > > >> >> >>> , >> > > > > > > > > > > > >> >> >>> > supporting >> > > > > > > > > > > > >> >> >>> > multiple data sources is not a trivial >> > > change. >> > > > I >> > > > > > > would >> > > > > > > > > > > strongly >> > > > > > > > > > > > >> >> >>> recommend >> > > > > > > > > > > > >> >> >>> > starting with a design document to >> > carefully >> > > > > > evaluate >> > > > > > > > the >> > > > > > > > > > > > >> >> architectural >> > > > > > > > > > > > >> >> >>> > implications and long term impact. >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query metrics seems >> > > > reasonable >> > > > > > > given >> > > > > > > > > the >> > > > > > > > > > > > >> current >> > > > > > > > > > > > >> >> >>> JDBC >> > > > > > > > > > > > >> >> >>> > based persistence model. That said, we >> may >> > > also >> > > > > > > > consider >> > > > > > > > > > > > >> alternative >> > > > > > > > > > > > >> >> >>> > storage models. For example, if we later >> > > adopt >> > > > a >> > > > > > time >> > > > > > > > > > series >> > > > > > > > > > > > >> system >> > > > > > > > > > > > >> >> >>> such as >> > > > > > > > > > > > >> >> >>> > Prometheus to store metrics, the query >> > model >> > > > and >> > > > > > > access >> > > > > > > > > > > > patterns >> > > > > > > > > > > > >> >> would >> > > > > > > > > > > > >> >> >>> be >> > > > > > > > > > > > >> >> >>> > fundamentally different. Designing the >> REST >> > > API >> > > > > > > without >> > > > > > > > > > > > >> considering >> > > > > > > > > > > > >> >> >>> these >> > > > > > > > > > > > >> >> >>> > potential evolutions may limit >> flexibility. >> > > I'd >> > > > > > > suggest >> > > > > > > > > to >> > > > > > > > > > > > start >> > > > > > > > > > > > >> >> with >> > > > > > > > > > > > >> >> >>> the >> > > > > > > > > > > > >> >> >>> > use case. >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > Yufei >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri >> > > > > > Bourlatchkov < >> > > > > > > > > > > > >> >> [email protected]> >> > > > > > > > > > > > >> >> >>> > wrote: >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > > Hi Anand, >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view... subject to >> discussion: >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to Polaris >> is >> > > > > > perfectly >> > > > > > > > > fine. >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific endpoint URIs >> and >> > > > > payloads >> > > > > > > > might >> > > > > > > > > > > > require >> > > > > > > > > > > > >> a >> > > > > > > > > > > > >> >> few >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a separate >> thread >> > > for >> > > > > that >> > > > > > > > might >> > > > > > > > > > be >> > > > > > > > > > > > >> best. >> > > > > > > > > > > > >> >> >>> > > Contributors commonly create Google >> Docs >> > > for >> > > > > new >> > > > > > > API >> > > > > > > > > > > > proposals >> > > > > > > > > > > > >> too >> > > > > > > > > > > > >> >> >>> (they >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as the email >> > > discussion >> > > > > > > > > > progresses). >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to try Markdown >> > > (with >> > > > > PRs) >> > > > > > > for >> > > > > > > > > > > > proposals >> > > > > > > > > > > > >> >> [1] >> > > > > > > > > > > > >> >> >>> ... >> > > > > > > > > > > > >> >> >>> > > feel free to give it a try if you are >> > > > > comfortable >> > > > > > > > with >> > > > > > > > > > > that. >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify whether you mean >> end >> > > > user >> > > > > > > > > utilities >> > > > > > > > > > or >> > > > > > > > > > > > >> admin >> > > > > > > > > > > > >> >> >>> user >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter case those >> might >> > > be >> > > > > more >> > > > > > > > > > suitable >> > > > > > > > > > > > for >> > > > > > > > > > > > >> the >> > > > > > > > > > > > >> >> >>> Admin >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO. >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > Why would these utilities be common >> with >> > > > > events? >> > > > > > > > IMHO, >> > > > > > > > > > > event >> > > > > > > > > > > > >> use >> > > > > > > > > > > > >> >> >>> cases >> > > > > > > > > > > > >> >> >>> > are >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit metrics. >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating metrics >> > > persistence >> > > > > from >> > > > > > > > > > MetaStore >> > > > > > > > > > > > >> >> >>> persistence >> > > > > > > > > > > > >> >> >>> > at >> > > > > > > > > > > > >> >> >>> > > the code level, so that they could be >> > mixed >> > > > and >> > > > > > > > matched >> > > > > > > > > > > > >> >> >>> independently. >> > > > > > > > > > > > >> >> >>> > The >> > > > > > > > > > > > >> >> >>> > > separate datasource question will >> become >> > a >> > > > > > > non-issue >> > > > > > > > > with >> > > > > > > > > > > > that >> > > > > > > > > > > > >> >> >>> approach, >> > > > > > > > > > > > >> >> >>> > I >> > > > > > > > > > > > >> >> >>> > > guess. >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > The rationale for separating scan >> metrics >> > > and >> > > > > > > > metastore >> > > > > > > > > > > > >> >> persistence >> > > > > > > > > > > > >> >> >>> is >> > > > > > > > > > > > >> >> >>> > that >> > > > > > > > > > > > >> >> >>> > > "cascading deletes" between them are >> > hardly >> > > > > ever >> > > > > > > > > > required. >> > > > > > > > > > > > >> >> >>> Furthermore, >> > > > > > > > > > > > >> >> >>> > the >> > > > > > > > > > > > >> >> >>> > > data and query patterns are very >> > different >> > > so >> > > > > > > > different >> > > > > > > > > > > > >> >> technologies >> > > > > > > > > > > > >> >> >>> > might >> > > > > > > > > > > > >> >> >>> > > be beneficial in each case. >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > [1] >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$ >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > Cheers, >> > > > > > > > > > > > >> >> >>> > > Dmitri. >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM Anand >> > Kumar >> > > > > > > Sankaran >> > > > > > > > > via >> > > > > > > > > > > dev >> > > > > > > > > > > > < >> > > > > > > > > > > > >> >> >>> > > [email protected]> wrote: >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is merged now. >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up features / >> work >> > > > > needed. >> > > > > > > > These >> > > > > > > > > > > were >> > > > > > > > > > > > >> all >> > > > > > > > > > > > >> >> >>> part of >> > > > > > > > > > > > >> >> >>> > > the >> > > > > > > > > > > > >> >> >>> > > > merged PR at some point in time and >> > were >> > > > > > removed >> > > > > > > to >> > > > > > > > > > > reduce >> > > > > > > > > > > > >> >> scope. >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what you think. >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > > 1. A REST API to paginate through >> > > table >> > > > > > > metrics. >> > > > > > > > > > This >> > > > > > > > > > > > >> will be >> > > > > > > > > > > > >> >> >>> > non-IRC >> > > > > > > > > > > > >> >> >>> > > > standard addition. >> > > > > > > > > > > > >> >> >>> > > > 2. Utilities for managing old >> > records, >> > > > > > should >> > > > > > > be >> > > > > > > > > > > common >> > > > > > > > > > > > >> with >> > > > > > > > > > > > >> >> >>> events. >> > > > > > > > > > > > >> >> >>> > > > There was some discussion that it >> > belongs >> > > > to >> > > > > > the >> > > > > > > > CLI. >> > > > > > > > > > > > >> >> >>> > > > 3. Separate datasource (metrics, >> > > events, >> > > > > > even >> > > > > > > > > other >> > > > > > > > > > > > >> tables?). >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else? >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > > - >> > > > > > > > > > > > >> >> >>> > > > Anand >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> >> > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
