Hi Alex, > it's not so much the event bus that needs benchmarking, but the dedicated event executor.
I'm also a bit confused by this, my understanding is that the executor mainly schedules listener execution. The actual throughput and resource consumption are largely determined by the listener implementation itself. A CloudWatch listener, JDBC listener, or custom listener can all have very different characteristics, and that's not really something Polaris can control. This is also why I'm not sure filtering at the dispatch stage addresses the scalability concern, which Yong and I explicitly asked for in this thread. If an event has already gone through the bus and executor queue before being dropped, it has already consumed framework resources. If the goal is to reduce listener load or avoid executor saturation, filtering closer to the producer is more effective. If the filtering framework's goal isn't performance or load reduction, I'm still not sure what problem it solves that listeners can't already handle. Yufei On Thu, Jun 18, 2026 at 7:27 AM Alexandre Dutra <[email protected]> wrote: > Hi Yufei, > > I appreciate you taking the time to review my PR. I have addressed > several of your points below: > > > Filters from the consumer side > > I haven't seen any filtering in the existing built-in listeners: they > process all events they receive. If you are suggesting we add it > there, my concern is that it would hinder the reusability and > composition of filters. > > > The PR seems to introduce a new filter on the dispatching route to > resolve the perf concern > > Indeed. But its purpose is to enhance the reusability and > composability of filters rather than to address performance. The idea > is to allow users to set up their own delivery pipelines through > configuration. > > > filtering should happen on the producer side before events hit the bus > > I have reservations about moving filtering to the producer side. Doing > so would force the HTTP request thread to handle filtering, whereas > the event bus was specifically designed to decouple request processing > from event processing. > > Today, events are processed on a dedicated executor. My PR simply adds > filtering to the last step, before the event is delivered to the > listener: > > producer (HTTP request thread) > -> event bus > -> dispatch (Vert.x event loop thread) > -> filters (NEW) -> sanitizer -> listener (event executor > thread) > > > we don't really know the limits of the event bus yet. > > That is true, particularly if we route metrics through it. BTW it's > not so much the event bus that needs benchmarking, but the dedicated > event executor. > > > If it performs well even under a large event volume, we may not need any > additional filtering infrastructure at all. > > I'm not sure I follow: if we find out the bus performs well, why would > we give up on filtering? > > > If the goal is to enable or disable metrics for a catalog, namespace, or > table, keeping that decision close to the entity makes lifecycle management > much simpler. > > I'm open to new ideas. How would you see this simplified workflow? > E.g. where would a per-catalog filtering happen and how would you > configure that? > > Thanks, > Alex > > On Wed, Jun 17, 2026 at 2:32 AM Yufei Gu <[email protected]> wrote: > > > > Hi Alex, > > > > Thanks for taking a stab at this. I briefly looked through PR 4773, and > I'm > > not sure if it's heading in the right direction. > > > > As I see it, the event flow is: *producers (REST endpoint delegators) → > > event bus → consumers (event listeners).* > > > > I see two levels of filters were available in the PR. Let's look at them > > separately: > > > > 1.* Filters from the consumer side (aws-cloudwatch): *It makes sense to > let > > each event listener implement the filters it needs. We may provide config > > for listeners shipped with Polaris. A custom listener is on its own to be > > implemented properly. > > 2. *Filters for the event bus: *The PR seems to introduce a new filter on > > the dispatching route to resolve the perf concern. However, if the goal > of > > filtering is to avoid a noisy neighbor problem, filtering should happen > on > > the producer side before events hit the bus. Otherwise, the event bus can > > still become saturated regardless of any downstream filtering. That said, > > we don't really know the limits of the event bus yet. If it performs well > > even under a large event volume, we may not need any additional filtering > > infrastructure at all. We should avoid premature optimization if > possible. > > I think we may need to benchmark the event bus first and understand its > > capabilities before jumping to a solution. > > > > For metrics-specific filtering, I think entity properties or a > > table-level/ns-level parameter may be a better fit than a generic > > expression-based event filter. If the goal is to enable or disable > metrics > > for a catalog, namespace, or table, keeping that decision close to the > > entity makes lifecycle management much simpler. Otherwise, operators need > > to keep filter expressions synchronized whenever entities are created, > > dropped, renamed, or moved. It also allows filtering to happen on the > > producer side before events reach the bus, which is more effective for > > reducing event bus load. > > > > Yufei > > > > > > On Mon, Jun 15, 2026 at 10:02 AM Alexandre Dutra <[email protected]> > wrote: > > > > > Hi Romain, > > > > > > GraalJS was introduced because of Ranger, but the long-term plan is to > > > remove it when Ranger will migrate to a sidecar-style deployment. > > > > > > Re: the factory: I still don't get it, sorry. But if you don't mind, > > > let's move this discussion to the PR. > > > > > > Thanks, > > > Alex > > > > > > On Mon, Jun 15, 2026 at 6:31 PM Romain Manni-Bucau > > > <[email protected]> wrote: > > > > > > > > Hmm, > > > > > > > > EL is fast if compiled so sometimes it can need (tomcat impl or RI > does) > > > a > > > > local work dir which is negative overall when ran as a container > until > > > you > > > > do it at build time (which defeats the feature I think). > > > > I got the same issue with CEL which is not able to do much until you > do > > > not > > > > use CEL but "my custom CEL with a lot of function" implementation so > > > > between both it is mainly an ecosystem choice, not much capabilities > > > IMHO. > > > > > > > > About js, graal-js is ~85M, polaris image is 715M...and graal js is > > > already > > > > there in 1.5.0, so maybe not that hurting after all ;) > > > > > > > > About the factory question: then JakartaElEventFilterFactory can be > > > totally > > > > deleted and the configuration injected directly at instantiation > which is > > > > easier for extension writer (a bit harder for SPI provider but this > is > > > the > > > > not important part IMHO). > > > > > > > > Le lun. 15 juin 2026 à 18:10, Alexandre Dutra <[email protected]> a > > > écrit : > > > > > > > > > Hey Romain, > > > > > > > > > > > Any planned default (in official docker image) alternative to EL? > > > > > > > > > > I don't think so. > > > > > > > > > > Re: javascript / graaljs, I'm indeed afraid the final image size > will > > > > > grow considerably, not to mention CPU overhead and dealing with all > > > > > the CVEs. The nice thing about Jakarta EL is that it is fast, > > > > > lightweight and already included in the server image. > > > > > > > > > > I think CEL would be a great fit, but CEL has met some pushback > > > > > because the historical impl is hosted under Project Nessie [1]. > That > > > > > said, the Google CEL impl [2] seems to have gained some momentum > > > > > lately, that could maybe be a good alternative (but I never tried > it). > > > > > > > > > > > I still wonder why there are these factories whereas the CDI > > > container > > > > > does handle it already well enough? > > > > > > > > > > If I'm getting your question right, it's because the same filter > can > > > > > be instantiated with N different configurations, e.g.: > > > > > > > > > > polaris.event-filter.filter1.type = jakarta-el > > > > > polaris.event-filter.filter2.type = jakarta-el > > > > > polaris.event-filter.filter1.include=attributes.catalog_name == > "dev" > > > > > polaris.event-filter.filter2.include=attributes.catalog_name == > "prod" > > > > > > > > > > Thanks, > > > > > Alex > > > > > > > > > > [1]: > https://lists.apache.org/thread/f28pp26s3mrgwpw2zrvq7snzr08z78sr > > > > > [2]: https://github.com/google/cel-java > > > > > > > > > > On Mon, Jun 15, 2026 at 5:32 PM Romain Manni-Bucau > > > > > <[email protected]> wrote: > > > > > > > > > > > > Hi Alexandre, > > > > > > > > > > > > Any planned default (in official docker image) alternative to > EL? got > > > > > > proven not sufficient as soon as you want some logic flow by my > past > > > > > > experience. > > > > > > Out of my head I can think to jython or javascript (using graaljs > > > > > > extension) but it is a bit fat-ty - but both are sandboxable > which > > > is key > > > > > > if expressions can be injected from end users at some point. > > > > > > Lua is way lighter to embed but language is less known and less > user > > > > > > friendly but can still be a compromise. > > > > > > > > > > > > Side question: I still wonder why there are these factories > whereas > > > the > > > > > CDI > > > > > > container does handle it already well enough? Would avoid to > have to > > > > > > specialize the default factory adding a new factory which is > always > > > nicer > > > > > > for vendors/integrators. > > > > > > > > > > > > Romain Manni-Bucau > > > > > > @rmannibucau <https://x.com/rmannibucau> | .NET Blog > > > > > > <https://dotnetbirdie.github.io/> | Blog < > > > https://rmannibucau.github.io/> > > > > > | Old > > > > > > Blog <http://rmannibucau.wordpress.com> | Github > > > > > > <https://github.com/rmannibucau> | LinkedIn > > > > > > <https://www.linkedin.com/in/rmannibucau> | Book > > > > > > < > > > > > > > > > https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064 > > > > > > > > > > > > Javaccino founder (Java/.NET service - contact via linkedin) > > > > > > > > > > > > > > > > > > Le lun. 15 juin 2026 à 15:45, Alexandre Dutra <[email protected]> > a > > > > > écrit : > > > > > > > > > > > > > Hi Yong, hi all, > > > > > > > > > > > > > > Since the ability to filter seems to be of concern, I went > ahead > > > and > > > > > > > implemented an EventFilter API with a first implementation > based on > > > > > > > Jakarta EL: > > > > > > > > > > > > > > https://github.com/apache/polaris/pull/4773 > > > > > > > > > > > > > > In the above PR, event filters form a composable chain: > > > > > > > > > > > > > > emitter -> filters (0..N) -> sanitizer (0..1) -> listener > > > > > > > > > > > > > > It's easy enough to create another EventFilter implementation > to do > > > > > > > some event sampling, as you suggested. > > > > > > > > > > > > > > (Note: the goal is to do the same with sanitizers and make them > > > 0..N > > > > > > > as well in the delivery pipeline.) > > > > > > > > > > > > > > Thanks, > > > > > > > Alex > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 8:13 AM Yong Zheng < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > Hello team, > > > > > > > > > > > > > > > > Thanks Yufei for the summary and for raising this discussion. > > > > > > > > > > > > > > > > One concern I have is around filtering. Without some form of > > > > > filtering or > > > > > > > > sampling, blindly writing every metrics event to logs (when > debug > > > > > logging > > > > > > > > is enabled) or persisting every metric to a backend could > > > introduce > > > > > > > > significant overhead. > > > > > > > > > > > > > > > > For Polaris itself, processing and forwarding large volumes > of > > > scan > > > > > > > metrics > > > > > > > > can consume resources that would otherwise be available for > > > serving > > > > > > > catalog > > > > > > > > requests. For deployments that persist metrics to a > database, the > > > > > > > > additional storage, indexing, and write workload can also > consume > > > > > compute > > > > > > > > resources that could be used for core catalog operations > instead. > > > > > > > > > > > > > > > > This is one of the reasons I think filtering should remain > part > > > of > > > > > the > > > > > > > > discussion regardless of whether metrics are delivered > through > > > JDBC > > > > > > > > persistence, the event framework, or another implementation. > > > > > High-volume > > > > > > > > scan metrics can easily generate far more traffic than many > > > > > deployments > > > > > > > > actually need to retain or analyze. > > > > > > > > > > > > > > > > I agree with the SPI direction, but I do not think it > addresses > > > the > > > > > > > > underlying scalability concern by itself. The SPI provides > > > > > flexibility in > > > > > > > > how metrics are handled, but the load is ultimately > determined > > > by the > > > > > > > > implementation behind it. Without some form of filtering or > > > sampling, > > > > > > > > high-volume scan metrics can still generate substantial load > > > > > regardless > > > > > > > of > > > > > > > > whether the implementation uses JDBC persistence, event > > > forwarding, > > > > > > > > logging, or something else. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Yong Zheng > > > > > > > > > > > > > > > > On Thu, Jun 11, 2026 at 10:30 PM EJ Wang < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Yufei, Alex, > > > > > > > > > > > > > > > > > > Thanks Yufei for writing this up, and Alex for spelling > out the > > > > > > > operational > > > > > > > > > concerns. My read is that both points are compatible if we > are > > > > > clear > > > > > > > about > > > > > > > > > the layering. > > > > > > > > > > > > > > > > > > I agree that Iceberg scan/commit metrics often behave like > > > > > structured > > > > > > > > > telemetry events: append-only, high-volume, usually > consumed > > > > > > > > > asynchronously, and often forwarded to external systems. > > > > > > > Events/listeners > > > > > > > > > are a natural fit for that kind of delivery path. > > > > > > > > > > > > > > > > > > I also agree with Alex that event delivery does not make > > > > > persistence, > > > > > > > > > filtering, retention, payload sizing, or performance free. > > > Those > > > > > are > > > > > > > real > > > > > > > > > concerns, especially for high-volume scan reports. > > > > > > > > > > > > > > > > > > The way I would reconcile these is to distinguish the > default > > > > > battery > > > > > > > from > > > > > > > > > extension implementations. > > > > > > > > > > > > > > > > > > The latest metrics sync alignment > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.k96s2xyqr5u1#heading=h.uvb454otvxc0 > > > > > > > > > > > > > > > > > > > was not that Polaris should pick JDBC, events, or external > > > > > telemetry > > > > > > > as the > > > > > > > > > one built-in metrics subsystem. It was closer to: Polaris > > > should > > > > > > > define a > > > > > > > > > clean metrics reporting/emitting boundary, ship a small > safe > > > > > default, > > > > > > > and > > > > > > > > > let deployments choose implementation paths behind that > > > boundary. > > > > > > > > > > > > > > > > > > Under that framing, I would not make event > > > > > > > > > forwarding/Prometheus/Grafana/custom routing the default > > > battery > > > > > > > itself. I > > > > > > > > > would frame it as a useful non-default extension > > > implementation of > > > > > the > > > > > > > > > metrics reporting/emitting path. > > > > > > > > > > > > > > > > > > Concretely, I think the split could be: > > > > > > > > > > > > > > > > > > 1. Polaris exposes a stable Iceberg metrics > reporting/emitting > > > > > SPI. > > > > > > > > > 2. The built-in default battery stays minimal: based on > the > > > latest > > > > > > > notes, > > > > > > > > > no-op or log-only is enough as the safe OSS default. > > > > > > > > > 3. Durable JDBC metrics storage is one named extension > > > > > implementation > > > > > > > of > > > > > > > > > that SPI, not part of core persistence. > > > > > > > > > 4. Event-based forwarding can be another named extension > > > > > > > implementation of > > > > > > > > > that SPI, where the listener/extension owns delivery, > > > filtering, > > > > > > > retention, > > > > > > > > > payload handling, and destination-specific behavior. > > > > > > > > > > > > > > > > > > That keeps the useful part of Yufei's proposal: deployments > > > that > > > > > want > > > > > > > > > Grafana/dashboard integration or custom telemetry routing > can > > > > > choose an > > > > > > > > > event/listener-based implementation. It also keeps Alex's > > > concerns > > > > > > > scoped > > > > > > > > > to the implementation that chooses that delivery model, > > > instead of > > > > > > > making > > > > > > > > > them requirements for every Polaris deployment or for the > > > built-in > > > > > > > default. > > > > > > > > > So I am generally +1 on exploring the event-forwarding > path, > > > with > > > > > the > > > > > > > > > layering caveat that I would treat it as an extension > > > > > implementation > > > > > > > of the > > > > > > > > > metrics reporting/emitting SPI, not as replacing the > default > > > > > battery or > > > > > > > > > collapsing metrics into core event persistence. > > > > > > > > > > > > > > > > > > Once that boundary is clear, which I'm pushing in PR4115 > > > > > > > > > < > > > > > > > > > > > > > > > > https://github.com/apache/polaris/pull/4115#pullrequestreview-4481873839 > > > > > > > > > >, > > > > > > > > > integrations become implementation choices rather than > > > > > architectural > > > > > > > > > changes. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > -ej > > > > > > > > > > > > > > > > > > On Thu, Jun 11, 2026 at 5:41 AM Alexandre Dutra < > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > listeners can already implement whatever filtering > logic > > > they > > > > > need > > > > > > > > > > > > > > > > > > > > True, but I think they would be reinventing the wheel > quite > > > > > often. > > > > > > > > > > There are some common filtering patterns such as > filtering by > > > > > > > catalog, > > > > > > > > > > namespace or table names or IDs. If we could provide this > > > filter > > > > > out > > > > > > > > > > of the box, that would be beneficial to many listeners. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 11, 2026 at 3:35 AM Yufei Gu < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Thanks all for the feedback! It seems we have some > initial > > > > > > > consensus > > > > > > > > > that > > > > > > > > > > > using the event framework for metrics delivery is a > > > reasonable > > > > > > > > > direction > > > > > > > > > > > worth exploring. Most of the discussion now appears to > be > > > > > around > > > > > > > impl > > > > > > > > > > > details and operational considerations. > > > > > > > > > > > > > > > > > > > > > > 1. Benchmarking is a great idea, using the existing > tool > > > makes > > > > > > > sense. I > > > > > > > > > > > don't see it as a blocker though. The volume of scan > > > metrics > > > > > > > should be > > > > > > > > > > > similar to, or even lower than, the volume of LoadTable > > > > > requests. > > > > > > > Some > > > > > > > > > > > clients may not send scan metrics at all. If we're > > > comfortable > > > > > > > > > supporting > > > > > > > > > > > LoadTable events, I'm not sure why metrics events would > > > > > require a > > > > > > > > > > > fundamentally different validation path, though > > > benchmarking > > > > > would > > > > > > > > > > > certainly help us tune the event bus and listener > > > > > configuration. > > > > > > > > > > > > > > > > > > > > > > 2. I agree that separating the datasource for event and > > > metrics > > > > > > > > > > persistence > > > > > > > > > > > is an active and worthwhile discussion. I think we > should > > > > > continue > > > > > > > that > > > > > > > > > > > work regardless of the direction we take here. > > > > > > > > > > > > > > > > > > > > > > 3. Agreed on evaluating payload sizes. That said, it > > > doesn't > > > > > seem > > > > > > > like > > > > > > > > > a > > > > > > > > > > > major concern to me given that we already support > larger > > > > > payloads > > > > > > > in > > > > > > > > > some > > > > > > > > > > > existing events. > > > > > > > > > > > > > > > > > > > > > > 4. Filtering is a valid use case. My thinking is that > > > custom > > > > > event > > > > > > > > > > > listeners can already implement whatever filtering > logic > > > they > > > > > > > need. I'm > > > > > > > > > > not > > > > > > > > > > > sure we need a generic filtering framework in Polaris > > > itself > > > > > yet, > > > > > > > but > > > > > > > > > I'm > > > > > > > > > > > open to further discussion if we find common > requirements > > > > > across > > > > > > > > > > > deployments. > > > > > > > > > > > > > > > > > > > > > > 5. Schema migration is a good point and something we > should > > > > > keep in > > > > > > > > > mind > > > > > > > > > > if > > > > > > > > > > > metrics are persisted. > > > > > > > > > > > > > > > > > > > > > > 6. I also agree with Dmitri that we can continue > improving > > > the > > > > > > > RDBMS > > > > > > > > > > schema > > > > > > > > > > > evolution story. That feels largely orthogonal to this > > > > > proposal, so > > > > > > > > > > perhaps > > > > > > > > > > > it's best discussed in a separate thread. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 10, 2026 at 12:56 PM Dmitri Bourlatchkov < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > +1 to all points from Alex's email. > > > > > > > > > > > > > > > > > > > > > > > > Re: Metrics Persistence I believe we ought to make > it as > > > > > smooth > > > > > > > as > > > > > > > > > > possible > > > > > > > > > > > > from the Polaris code maintenance perspective. > > > Therefore, I > > > > > > > propose > > > > > > > > > > > > starting the work to isolate the existing metrics > schema > > > > > from the > > > > > > > > > > MetaStore > > > > > > > > > > > > schema in parallel with the event bus work. I think > it > > > will > > > > > be > > > > > > > > > > beneficial > > > > > > > > > > > > in its own right, regardless of how the event bus > work > > > > > > > progresses. > > > > > > > > > > > > > > > > > > > > > > > > PR [4397] is but the first step in that direction. > > > > > > > > > > > > > > > > > > > > > > > > Side note: we probably do not need to copy the whole > > > schema > > > > > SQL > > > > > > > file > > > > > > > > > on > > > > > > > > > > > > every revision, but I'm contemplating starting a > separate > > > > > thread > > > > > > > on > > > > > > > > > > that. > > > > > > > > > > > > > > > > > > > > > > > > Once a separate metrics schema is established, I > think it > > > > > will be > > > > > > > > > > natural > > > > > > > > > > > > to also allow it to be on a different JDBC DataSource > > > than > > > > > the > > > > > > > > > > MetaStore > > > > > > > > > > > > schema. > > > > > > > > > > > > > > > > > > > > > > > > If the event bus work is successful, JDBC Metrics > > > > > Persistence can > > > > > > > > > > become > > > > > > > > > > > > one of possibly many consumers for metrics events. > > > > > > > > > > > > > > > > > > > > > > > > With this approach, it should also be possible to > write > > > > > metrics > > > > > > > to > > > > > > > > > the > > > > > > > > > > > > database in batches. IIRC, Venkateshwaran brought > this > > > point > > > > > up > > > > > > > in > > > > > > > > > the > > > > > > > > > > > > latest Metrics Sync meeting. > > > > > > > > > > > > > > > > > > > > > > > > Metrics filtering can probably progress in parallel > too. > > > I > > > > > think > > > > > > > it > > > > > > > > > is > > > > > > > > > > a > > > > > > > > > > > > useful feature. > > > > > > > > > > > > > > > > > > > > > > > > [4397] https://github.com/apache/polaris/pull/4397 > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 10, 2026 at 9:56 AM Alexandre Dutra < > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yufei, > > > > > > > > > > > > > > > > > > > > > > > > > > The proposal to leverage the events subsystem for > > > metrics > > > > > > > delivery > > > > > > > > > is > > > > > > > > > > > > > quite appealing, though it requires a thorough > > > evaluation > > > > > > > regarding > > > > > > > > > > > > > potential performance overhead. > > > > > > > > > > > > > > > > > > > > > > > > > > My primary considerations are as follows: > > > > > > > > > > > > > > > > > > > > > > > > > > 1) Given that scan reports can trigger a high > volume of > > > > > > > events, we > > > > > > > > > > > > > should conduct rigorous testing, potentially using > the > > > > > Polaris > > > > > > > > > > > > > benchmark tool. We need to determine what's the > right > > > > > > > configuration > > > > > > > > > > > > > for the event bus and for the event listener > executor. > > > > > > > > > > > > > > > > > > > > > > > > > > 2) While the events subsystem handles dispatch and > > > delivery > > > > > > > > > natively, > > > > > > > > > > > > > it doesn't give persistence for free. My > recollection > > > is > > > > > that > > > > > > > we > > > > > > > > > were > > > > > > > > > > > > > pursuing the idea of a metrics persistence system > with > > > a > > > > > unique > > > > > > > > > > schema > > > > > > > > > > > > > and possibly a separate datasource, a process > > > initiated by > > > > > a > > > > > > > > > recently > > > > > > > > > > > > > merged PR [1]. Is that still the case? Furthermore, > > > we'd > > > > > need > > > > > > > to > > > > > > > > > > > > > implement data retention and purging, including > for the > > > > > current > > > > > > > > > > events > > > > > > > > > > > > > table [2]. > > > > > > > > > > > > > > > > > > > > > > > > > > 3) If we consider the events table for metrics > > > storage, we > > > > > must > > > > > > > > > > > > > evaluate average payload sizes. Although a PR [3] > was > > > > > > > introduced to > > > > > > > > > > > > > prune large payloads (such as table metadata), this > > > > > > > functionality > > > > > > > > > is > > > > > > > > > > > > > still in its early stages and will evolve. Similar > > > pruning > > > > > > > would be > > > > > > > > > > > > > necessary for metrics reports if they are big. > > > > > > > > > > > > > > > > > > > > > > > > > > 4) As Yong suggested [4], we may still require more > > > > > > > sophisticated > > > > > > > > > > > > > metrics filtering. The events subsystem currently > only > > > > > allows > > > > > > > > > > > > > filtering by event type or event category, which > may > > > not be > > > > > > > > > granular > > > > > > > > > > > > > enough for our needs (as of today, it would allow > only > > > to > > > > > > > > > distinguish > > > > > > > > > > > > > scan vs metrics reports). In that regard, I would > > > welcome > > > > > the > > > > > > > > > > > > > opportunity to implement a generic EventFilter > > > interface > > > > > with a > > > > > > > > > > > > > default implementation based on CEL. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: https://github.com/apache/polaris/pull/4397 > > > > > > > > > > > > > [2]: > > > > > > > > > > > > > https://lists.apache.org/thread/krmddx8myov926sd0mbh4ogy8sdgrfgq > > > > > > > > > > > > > [3]: https://github.com/apache/polaris/pull/4225 > > > > > > > > > > > > > [4]: > > > > > > > > > > > > > https://lists.apache.org/thread/ogskc1szctkg5n0tdj0cm3pfkowcwx4z > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 10, 2026 at 2:04 AM Yufei Gu < > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've been thinking about how Polaris should > support > > > > > Iceberg > > > > > > > scan > > > > > > > > > > and > > > > > > > > > > > > > commit > > > > > > > > > > > > > > metrics. A few challenges have come up in recent > > > > > discussions: > > > > > > > > > > > > > > 1. Sync metrics persistence chokes Polaris > > > persistence > > > > > due > > > > > > > to the > > > > > > > > > > high > > > > > > > > > > > > > > volume of scan metrics [3]. > > > > > > > > > > > > > > 2. We spent considerable time figuring out the > > > metrics > > > > > > > > > persistence, > > > > > > > > > > > > > > including the schema, SPIs, REST APIs [4]. > > > > > > > > > > > > > > 3. Metric filtering remains a challenge [1]. > > > > > > > > > > > > > > 4. We need to figure out how to purge metrics > because > > > > > they > > > > > > > keep > > > > > > > > > > growing > > > > > > > > > > > > > [2]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looking at these challenges, most of them are not > > > really > > > > > > > metrics > > > > > > > > > > > > > problems. > > > > > > > > > > > > > > They are transport, delivery, retention, and > > > lifecycle > > > > > > > problems > > > > > > > > > > that > > > > > > > > > > > > the > > > > > > > > > > > > > > existing event framework already addresses. I'd > like > > > to > > > > > > > propose > > > > > > > > > > using > > > > > > > > > > > > the > > > > > > > > > > > > > > event system to facilitate the current use cases > of > > > > > Iceberg > > > > > > > scan > > > > > > > > > > and > > > > > > > > > > > > > commit > > > > > > > > > > > > > > metrics rather than introducing a separate > Polaris > > > > > metrics > > > > > > > > > > subsystem. > > > > > > > > > > > > The > > > > > > > > > > > > > > metrics for current use cases are fundamentally > > > events > > > > > with > > > > > > > > > > structured > > > > > > > > > > > > > > telemetry attached. They are append only, > generated > > > by > > > > > IRC > > > > > > > > > > endpoints, > > > > > > > > > > > > > > typically consumed asynchronously, and often > > > forwarded to > > > > > > > > > external > > > > > > > > > > > > > systems. > > > > > > > > > > > > > > Since Polaris already needs to support them as > part > > > of > > > > > IRC, > > > > > > > > > > treating > > > > > > > > > > > > them > > > > > > > > > > > > > > as event types seems like a natural fit. > > > > > > > > > > > > > > > > > > > > > > > > > > > > More importantly, I think Polaris should remain a > > > catalog > > > > > > > service > > > > > > > > > > and > > > > > > > > > > > > > > telemetry producer rather than a metrics > warehouse. > > > > > Instead > > > > > > > of > > > > > > > > > > > > > introducing > > > > > > > > > > > > > > a dedicated metrics subsystem along with storage, > > > > > retention, > > > > > > > > > > query, and > > > > > > > > > > > > > > scaling concerns, we could build on the existing > > > event > > > > > > > framework: > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Emit them through the existing event > mechanism. > > > We > > > > > will > > > > > > > do > > > > > > > > > > that > > > > > > > > > > > > > anyway > > > > > > > > > > > > > > given it's an IRC endpoint. > > > > > > > > > > > > > > - Let custom event listeners route them to the > > > > > > > destination of > > > > > > > > > > > > choice, > > > > > > > > > > > > > > such as Prometheus, Grafana, RDBMSs, or other > > > systems. > > > > > > > > > > > > > > - Reuse the existing event lifecycle, > retention, > > > and > > > > > > > delivery > > > > > > > > > > > > models. > > > > > > > > > > > > > If > > > > > > > > > > > > > > temporary persistence is still required, the > > > existing > > > > > > > event > > > > > > > > > > table > > > > > > > > > > > > can > > > > > > > > > > > > > serve > > > > > > > > > > > > > > that purpose. The payload size is manageable > given > > > > > that we > > > > > > > > > have > > > > > > > > > > put > > > > > > > > > > > > > the > > > > > > > > > > > > > > loadTable/LoadView response in events. > > > > > > > > > > > > > > > > > > > > > > > > > > > > This approach also gives deployments flexibility > to > > > > > filter, > > > > > > > > > > sample, or > > > > > > > > > > > > > > redirect high volume scan metrics without Polaris > > > needing > > > > > > > backend > > > > > > > > > > > > > specific > > > > > > > > > > > > > > metric storage behavior. For example, event > > > listeners can > > > > > > > choose > > > > > > > > > > which > > > > > > > > > > > > > > metric events to process. We don't need to > implement > > > > > metric > > > > > > > > > > filtering > > > > > > > > > > > > > logic > > > > > > > > > > > > > > [1]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > In short, my proposal is: Events provide the > > > transport > > > > > and > > > > > > > > > > lifecycle > > > > > > > > > > > > > > mechanism, while downstream metrics systems > remain > > > > > > > responsible > > > > > > > > > for > > > > > > > > > > > > > storage, > > > > > > > > > > > > > > querying, aggregation, and visualization. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Curious what others think. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > https://lists.apache.org/thread/ogskc1szctkg5n0tdj0cm3pfkowcwx4z > > > > > > > > > > > > > > 2. > > > > > > > > > > > > > https://lists.apache.org/thread/5nst0f2ygnl2gj3j910q7m8nk2fvokc7 > > > > > > > > > > > > > > 3. > > > > > > > > > > > > > https://lists.apache.org/thread/zp2rvsdkq3mb46722o0hfl0zh7kdqyr8 > > > > > > > > > > > > > > 4. > > > > > > > > > > > > > https://lists.apache.org/thread/qj1y7cw4dygcnczmymdwkfkp4ysq41ts > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
