Hi Yufei,
I appreciate you taking the time to review my PR. I have addressed
several of your points below:
> Filters from the consumer side
I haven't seen any filtering in the existing built-in listeners: they
process all events they receive. If you are suggesting we add it
there, my concern is that it would hinder the reusability and
composition of filters.
> The PR seems to introduce a new filter on the dispatching route to resolve
> the perf concern
Indeed. But its purpose is to enhance the reusability and
composability of filters rather than to address performance. The idea
is to allow users to set up their own delivery pipelines through
configuration.
> filtering should happen on the producer side before events hit the bus
I have reservations about moving filtering to the producer side. Doing
so would force the HTTP request thread to handle filtering, whereas
the event bus was specifically designed to decouple request processing
from event processing.
Today, events are processed on a dedicated executor. My PR simply adds
filtering to the last step, before the event is delivered to the
listener:
producer (HTTP request thread)
-> event bus
-> dispatch (Vert.x event loop thread)
-> filters (NEW) -> sanitizer -> listener (event executor thread)
> we don't really know the limits of the event bus yet.
That is true, particularly if we route metrics through it. BTW it's
not so much the event bus that needs benchmarking, but the dedicated
event executor.
> If it performs well even under a large event volume, we may not need any
> additional filtering infrastructure at all.
I'm not sure I follow: if we find out the bus performs well, why would
we give up on filtering?
> If the goal is to enable or disable metrics for a catalog, namespace, or
> table, keeping that decision close to the entity makes lifecycle management
> much simpler.
I'm open to new ideas. How would you see this simplified workflow?
E.g. where would a per-catalog filtering happen and how would you
configure that?
Thanks,
Alex
On Wed, Jun 17, 2026 at 2:32 AM Yufei Gu <[email protected]> wrote:
>
> Hi Alex,
>
> Thanks for taking a stab at this. I briefly looked through PR 4773, and I'm
> not sure if it's heading in the right direction.
>
> As I see it, the event flow is: *producers (REST endpoint delegators) →
> event bus → consumers (event listeners).*
>
> I see two levels of filters were available in the PR. Let's look at them
> separately:
>
> 1.* Filters from the consumer side (aws-cloudwatch): *It makes sense to let
> each event listener implement the filters it needs. We may provide config
> for listeners shipped with Polaris. A custom listener is on its own to be
> implemented properly.
> 2. *Filters for the event bus: *The PR seems to introduce a new filter on
> the dispatching route to resolve the perf concern. However, if the goal of
> filtering is to avoid a noisy neighbor problem, filtering should happen on
> the producer side before events hit the bus. Otherwise, the event bus can
> still become saturated regardless of any downstream filtering. That said,
> we don't really know the limits of the event bus yet. If it performs well
> even under a large event volume, we may not need any additional filtering
> infrastructure at all. We should avoid premature optimization if possible.
> I think we may need to benchmark the event bus first and understand its
> capabilities before jumping to a solution.
>
> For metrics-specific filtering, I think entity properties or a
> table-level/ns-level parameter may be a better fit than a generic
> expression-based event filter. If the goal is to enable or disable metrics
> for a catalog, namespace, or table, keeping that decision close to the
> entity makes lifecycle management much simpler. Otherwise, operators need
> to keep filter expressions synchronized whenever entities are created,
> dropped, renamed, or moved. It also allows filtering to happen on the
> producer side before events reach the bus, which is more effective for
> reducing event bus load.
>
> Yufei
>
>
> On Mon, Jun 15, 2026 at 10:02 AM Alexandre Dutra <[email protected]> wrote:
>
> > Hi Romain,
> >
> > GraalJS was introduced because of Ranger, but the long-term plan is to
> > remove it when Ranger will migrate to a sidecar-style deployment.
> >
> > Re: the factory: I still don't get it, sorry. But if you don't mind,
> > let's move this discussion to the PR.
> >
> > Thanks,
> > Alex
> >
> > On Mon, Jun 15, 2026 at 6:31 PM Romain Manni-Bucau
> > <[email protected]> wrote:
> > >
> > > Hmm,
> > >
> > > EL is fast if compiled so sometimes it can need (tomcat impl or RI does)
> > a
> > > local work dir which is negative overall when ran as a container until
> > you
> > > do it at build time (which defeats the feature I think).
> > > I got the same issue with CEL which is not able to do much until you do
> > not
> > > use CEL but "my custom CEL with a lot of function" implementation so
> > > between both it is mainly an ecosystem choice, not much capabilities
> > IMHO.
> > >
> > > About js, graal-js is ~85M, polaris image is 715M...and graal js is
> > already
> > > there in 1.5.0, so maybe not that hurting after all ;)
> > >
> > > About the factory question: then JakartaElEventFilterFactory can be
> > totally
> > > deleted and the configuration injected directly at instantiation which is
> > > easier for extension writer (a bit harder for SPI provider but this is
> > the
> > > not important part IMHO).
> > >
> > > Le lun. 15 juin 2026 à 18:10, Alexandre Dutra <[email protected]> a
> > écrit :
> > >
> > > > Hey Romain,
> > > >
> > > > > Any planned default (in official docker image) alternative to EL?
> > > >
> > > > I don't think so.
> > > >
> > > > Re: javascript / graaljs, I'm indeed afraid the final image size will
> > > > grow considerably, not to mention CPU overhead and dealing with all
> > > > the CVEs. The nice thing about Jakarta EL is that it is fast,
> > > > lightweight and already included in the server image.
> > > >
> > > > I think CEL would be a great fit, but CEL has met some pushback
> > > > because the historical impl is hosted under Project Nessie [1]. That
> > > > said, the Google CEL impl [2] seems to have gained some momentum
> > > > lately, that could maybe be a good alternative (but I never tried it).
> > > >
> > > > > I still wonder why there are these factories whereas the CDI
> > container
> > > > does handle it already well enough?
> > > >
> > > > If I'm getting your question right, it's because the same filter can
> > > > be instantiated with N different configurations, e.g.:
> > > >
> > > > polaris.event-filter.filter1.type = jakarta-el
> > > > polaris.event-filter.filter2.type = jakarta-el
> > > > polaris.event-filter.filter1.include=attributes.catalog_name == "dev"
> > > > polaris.event-filter.filter2.include=attributes.catalog_name == "prod"
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > > [1]: https://lists.apache.org/thread/f28pp26s3mrgwpw2zrvq7snzr08z78sr
> > > > [2]: https://github.com/google/cel-java
> > > >
> > > > On Mon, Jun 15, 2026 at 5:32 PM Romain Manni-Bucau
> > > > <[email protected]> wrote:
> > > > >
> > > > > Hi Alexandre,
> > > > >
> > > > > Any planned default (in official docker image) alternative to EL? got
> > > > > proven not sufficient as soon as you want some logic flow by my past
> > > > > experience.
> > > > > Out of my head I can think to jython or javascript (using graaljs
> > > > > extension) but it is a bit fat-ty - but both are sandboxable which
> > is key
> > > > > if expressions can be injected from end users at some point.
> > > > > Lua is way lighter to embed but language is less known and less user
> > > > > friendly but can still be a compromise.
> > > > >
> > > > > Side question: I still wonder why there are these factories whereas
> > the
> > > > CDI
> > > > > container does handle it already well enough? Would avoid to have to
> > > > > specialize the default factory adding a new factory which is always
> > nicer
> > > > > for vendors/integrators.
> > > > >
> > > > > Romain Manni-Bucau
> > > > > @rmannibucau <https://x.com/rmannibucau> | .NET Blog
> > > > > <https://dotnetbirdie.github.io/> | Blog <
> > https://rmannibucau.github.io/>
> > > > | Old
> > > > > Blog <http://rmannibucau.wordpress.com> | Github
> > > > > <https://github.com/rmannibucau> | LinkedIn
> > > > > <https://www.linkedin.com/in/rmannibucau> | Book
> > > > > <
> > > >
> > https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064
> > > > >
> > > > > Javaccino founder (Java/.NET service - contact via linkedin)
> > > > >
> > > > >
> > > > > Le lun. 15 juin 2026 à 15:45, Alexandre Dutra <[email protected]> a
> > > > écrit :
> > > > >
> > > > > > Hi Yong, hi all,
> > > > > >
> > > > > > Since the ability to filter seems to be of concern, I went ahead
> > and
> > > > > > implemented an EventFilter API with a first implementation based on
> > > > > > Jakarta EL:
> > > > > >
> > > > > > https://github.com/apache/polaris/pull/4773
> > > > > >
> > > > > > In the above PR, event filters form a composable chain:
> > > > > >
> > > > > > emitter -> filters (0..N) -> sanitizer (0..1) -> listener
> > > > > >
> > > > > > It's easy enough to create another EventFilter implementation to do
> > > > > > some event sampling, as you suggested.
> > > > > >
> > > > > > (Note: the goal is to do the same with sanitizers and make them
> > 0..N
> > > > > > as well in the delivery pipeline.)
> > > > > >
> > > > > > Thanks,
> > > > > > Alex
> > > > > >
> > > > > > On Mon, Jun 15, 2026 at 8:13 AM Yong Zheng <
> > [email protected]>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hello team,
> > > > > > >
> > > > > > > Thanks Yufei for the summary and for raising this discussion.
> > > > > > >
> > > > > > > One concern I have is around filtering. Without some form of
> > > > filtering or
> > > > > > > sampling, blindly writing every metrics event to logs (when debug
> > > > logging
> > > > > > > is enabled) or persisting every metric to a backend could
> > introduce
> > > > > > > significant overhead.
> > > > > > >
> > > > > > > For Polaris itself, processing and forwarding large volumes of
> > scan
> > > > > > metrics
> > > > > > > can consume resources that would otherwise be available for
> > serving
> > > > > > catalog
> > > > > > > requests. For deployments that persist metrics to a database, the
> > > > > > > additional storage, indexing, and write workload can also consume
> > > > compute
> > > > > > > resources that could be used for core catalog operations instead.
> > > > > > >
> > > > > > > This is one of the reasons I think filtering should remain part
> > of
> > > > the
> > > > > > > discussion regardless of whether metrics are delivered through
> > JDBC
> > > > > > > persistence, the event framework, or another implementation.
> > > > High-volume
> > > > > > > scan metrics can easily generate far more traffic than many
> > > > deployments
> > > > > > > actually need to retain or analyze.
> > > > > > >
> > > > > > > I agree with the SPI direction, but I do not think it addresses
> > the
> > > > > > > underlying scalability concern by itself. The SPI provides
> > > > flexibility in
> > > > > > > how metrics are handled, but the load is ultimately determined
> > by the
> > > > > > > implementation behind it. Without some form of filtering or
> > sampling,
> > > > > > > high-volume scan metrics can still generate substantial load
> > > > regardless
> > > > > > of
> > > > > > > whether the implementation uses JDBC persistence, event
> > forwarding,
> > > > > > > logging, or something else.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yong Zheng
> > > > > > >
> > > > > > > On Thu, Jun 11, 2026 at 10:30 PM EJ Wang <
> > > > [email protected]
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Yufei, Alex,
> > > > > > > >
> > > > > > > > Thanks Yufei for writing this up, and Alex for spelling out the
> > > > > > operational
> > > > > > > > concerns. My read is that both points are compatible if we are
> > > > clear
> > > > > > about
> > > > > > > > the layering.
> > > > > > > >
> > > > > > > > I agree that Iceberg scan/commit metrics often behave like
> > > > structured
> > > > > > > > telemetry events: append-only, high-volume, usually consumed
> > > > > > > > asynchronously, and often forwarded to external systems.
> > > > > > Events/listeners
> > > > > > > > are a natural fit for that kind of delivery path.
> > > > > > > >
> > > > > > > > I also agree with Alex that event delivery does not make
> > > > persistence,
> > > > > > > > filtering, retention, payload sizing, or performance free.
> > Those
> > > > are
> > > > > > real
> > > > > > > > concerns, especially for high-volume scan reports.
> > > > > > > >
> > > > > > > > The way I would reconcile these is to distinguish the default
> > > > battery
> > > > > > from
> > > > > > > > extension implementations.
> > > > > > > >
> > > > > > > > The latest metrics sync alignment
> > > > > > > > <
> > > > > > > >
> > > > > >
> > > >
> > https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?pli=1&tab=t.k96s2xyqr5u1#heading=h.uvb454otvxc0
> > > > > > > > >
> > > > > > > > was not that Polaris should pick JDBC, events, or external
> > > > telemetry
> > > > > > as the
> > > > > > > > one built-in metrics subsystem. It was closer to: Polaris
> > should
> > > > > > define a
> > > > > > > > clean metrics reporting/emitting boundary, ship a small safe
> > > > default,
> > > > > > and
> > > > > > > > let deployments choose implementation paths behind that
> > boundary.
> > > > > > > >
> > > > > > > > Under that framing, I would not make event
> > > > > > > > forwarding/Prometheus/Grafana/custom routing the default
> > battery
> > > > > > itself. I
> > > > > > > > would frame it as a useful non-default extension
> > implementation of
> > > > the
> > > > > > > > metrics reporting/emitting path.
> > > > > > > >
> > > > > > > > Concretely, I think the split could be:
> > > > > > > >
> > > > > > > > 1. Polaris exposes a stable Iceberg metrics reporting/emitting
> > > > SPI.
> > > > > > > > 2. The built-in default battery stays minimal: based on the
> > latest
> > > > > > notes,
> > > > > > > > no-op or log-only is enough as the safe OSS default.
> > > > > > > > 3. Durable JDBC metrics storage is one named extension
> > > > implementation
> > > > > > of
> > > > > > > > that SPI, not part of core persistence.
> > > > > > > > 4. Event-based forwarding can be another named extension
> > > > > > implementation of
> > > > > > > > that SPI, where the listener/extension owns delivery,
> > filtering,
> > > > > > retention,
> > > > > > > > payload handling, and destination-specific behavior.
> > > > > > > >
> > > > > > > > That keeps the useful part of Yufei's proposal: deployments
> > that
> > > > want
> > > > > > > > Grafana/dashboard integration or custom telemetry routing can
> > > > choose an
> > > > > > > > event/listener-based implementation. It also keeps Alex's
> > concerns
> > > > > > scoped
> > > > > > > > to the implementation that chooses that delivery model,
> > instead of
> > > > > > making
> > > > > > > > them requirements for every Polaris deployment or for the
> > built-in
> > > > > > default.
> > > > > > > > So I am generally +1 on exploring the event-forwarding path,
> > with
> > > > the
> > > > > > > > layering caveat that I would treat it as an extension
> > > > implementation
> > > > > > of the
> > > > > > > > metrics reporting/emitting SPI, not as replacing the default
> > > > battery or
> > > > > > > > collapsing metrics into core event persistence.
> > > > > > > >
> > > > > > > > Once that boundary is clear, which I'm pushing in PR4115
> > > > > > > > <
> > > > > >
> > > >
> > https://github.com/apache/polaris/pull/4115#pullrequestreview-4481873839
> > > > > > > > >,
> > > > > > > > integrations become implementation choices rather than
> > > > architectural
> > > > > > > > changes.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > -ej
> > > > > > > >
> > > > > > > > On Thu, Jun 11, 2026 at 5:41 AM Alexandre Dutra <
> > [email protected]
> > > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > > listeners can already implement whatever filtering logic
> > they
> > > > need
> > > > > > > > >
> > > > > > > > > True, but I think they would be reinventing the wheel quite
> > > > often.
> > > > > > > > > There are some common filtering patterns such as filtering by
> > > > > > catalog,
> > > > > > > > > namespace or table names or IDs. If we could provide this
> > filter
> > > > out
> > > > > > > > > of the box, that would be beneficial to many listeners.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Alex
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jun 11, 2026 at 3:35 AM Yufei Gu <
> > [email protected]>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Thanks all for the feedback! It seems we have some initial
> > > > > > consensus
> > > > > > > > that
> > > > > > > > > > using the event framework for metrics delivery is a
> > reasonable
> > > > > > > > direction
> > > > > > > > > > worth exploring. Most of the discussion now appears to be
> > > > around
> > > > > > impl
> > > > > > > > > > details and operational considerations.
> > > > > > > > > >
> > > > > > > > > > 1. Benchmarking is a great idea, using the existing tool
> > makes
> > > > > > sense. I
> > > > > > > > > > don't see it as a blocker though. The volume of scan
> > metrics
> > > > > > should be
> > > > > > > > > > similar to, or even lower than, the volume of LoadTable
> > > > requests.
> > > > > > Some
> > > > > > > > > > clients may not send scan metrics at all. If we're
> > comfortable
> > > > > > > > supporting
> > > > > > > > > > LoadTable events, I'm not sure why metrics events would
> > > > require a
> > > > > > > > > > fundamentally different validation path, though
> > benchmarking
> > > > would
> > > > > > > > > > certainly help us tune the event bus and listener
> > > > configuration.
> > > > > > > > > >
> > > > > > > > > > 2. I agree that separating the datasource for event and
> > metrics
> > > > > > > > > persistence
> > > > > > > > > > is an active and worthwhile discussion. I think we should
> > > > continue
> > > > > > that
> > > > > > > > > > work regardless of the direction we take here.
> > > > > > > > > >
> > > > > > > > > > 3. Agreed on evaluating payload sizes. That said, it
> > doesn't
> > > > seem
> > > > > > like
> > > > > > > > a
> > > > > > > > > > major concern to me given that we already support larger
> > > > payloads
> > > > > > in
> > > > > > > > some
> > > > > > > > > > existing events.
> > > > > > > > > >
> > > > > > > > > > 4. Filtering is a valid use case. My thinking is that
> > custom
> > > > event
> > > > > > > > > > listeners can already implement whatever filtering logic
> > they
> > > > > > need. I'm
> > > > > > > > > not
> > > > > > > > > > sure we need a generic filtering framework in Polaris
> > itself
> > > > yet,
> > > > > > but
> > > > > > > > I'm
> > > > > > > > > > open to further discussion if we find common requirements
> > > > across
> > > > > > > > > > deployments.
> > > > > > > > > >
> > > > > > > > > > 5. Schema migration is a good point and something we should
> > > > keep in
> > > > > > > > mind
> > > > > > > > > if
> > > > > > > > > > metrics are persisted.
> > > > > > > > > >
> > > > > > > > > > 6. I also agree with Dmitri that we can continue improving
> > the
> > > > > > RDBMS
> > > > > > > > > schema
> > > > > > > > > > evolution story. That feels largely orthogonal to this
> > > > proposal, so
> > > > > > > > > perhaps
> > > > > > > > > > it's best discussed in a separate thread.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Yufei
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Jun 10, 2026 at 12:56 PM Dmitri Bourlatchkov <
> > > > > > [email protected]
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi All,
> > > > > > > > > > >
> > > > > > > > > > > +1 to all points from Alex's email.
> > > > > > > > > > >
> > > > > > > > > > > Re: Metrics Persistence I believe we ought to make it as
> > > > smooth
> > > > > > as
> > > > > > > > > possible
> > > > > > > > > > > from the Polaris code maintenance perspective.
> > Therefore, I
> > > > > > propose
> > > > > > > > > > > starting the work to isolate the existing metrics schema
> > > > from the
> > > > > > > > > MetaStore
> > > > > > > > > > > schema in parallel with the event bus work. I think it
> > will
> > > > be
> > > > > > > > > beneficial
> > > > > > > > > > > in its own right, regardless of how the event bus work
> > > > > > progresses.
> > > > > > > > > > >
> > > > > > > > > > > PR [4397] is but the first step in that direction.
> > > > > > > > > > >
> > > > > > > > > > > Side note: we probably do not need to copy the whole
> > schema
> > > > SQL
> > > > > > file
> > > > > > > > on
> > > > > > > > > > > every revision, but I'm contemplating starting a separate
> > > > thread
> > > > > > on
> > > > > > > > > that.
> > > > > > > > > > >
> > > > > > > > > > > Once a separate metrics schema is established, I think it
> > > > will be
> > > > > > > > > natural
> > > > > > > > > > > to also allow it to be on a different JDBC DataSource
> > than
> > > > the
> > > > > > > > > MetaStore
> > > > > > > > > > > schema.
> > > > > > > > > > >
> > > > > > > > > > > If the event bus work is successful, JDBC Metrics
> > > > Persistence can
> > > > > > > > > become
> > > > > > > > > > > one of possibly many consumers for metrics events.
> > > > > > > > > > >
> > > > > > > > > > > With this approach, it should also be possible to write
> > > > metrics
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > database in batches. IIRC, Venkateshwaran brought this
> > point
> > > > up
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > latest Metrics Sync meeting.
> > > > > > > > > > >
> > > > > > > > > > > Metrics filtering can probably progress in parallel too.
> > I
> > > > think
> > > > > > it
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > > > useful feature.
> > > > > > > > > > >
> > > > > > > > > > > [4397] https://github.com/apache/polaris/pull/4397
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jun 10, 2026 at 9:56 AM Alexandre Dutra <
> > > > > > [email protected]>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Yufei,
> > > > > > > > > > > >
> > > > > > > > > > > > The proposal to leverage the events subsystem for
> > metrics
> > > > > > delivery
> > > > > > > > is
> > > > > > > > > > > > quite appealing, though it requires a thorough
> > evaluation
> > > > > > regarding
> > > > > > > > > > > > potential performance overhead.
> > > > > > > > > > > >
> > > > > > > > > > > > My primary considerations are as follows:
> > > > > > > > > > > >
> > > > > > > > > > > > 1) Given that scan reports can trigger a high volume of
> > > > > > events, we
> > > > > > > > > > > > should conduct rigorous testing, potentially using the
> > > > Polaris
> > > > > > > > > > > > benchmark tool. We need to determine what's the right
> > > > > > configuration
> > > > > > > > > > > > for the event bus and for the event listener executor.
> > > > > > > > > > > >
> > > > > > > > > > > > 2) While the events subsystem handles dispatch and
> > delivery
> > > > > > > > natively,
> > > > > > > > > > > > it doesn't give persistence for free. My recollection
> > is
> > > > that
> > > > > > we
> > > > > > > > were
> > > > > > > > > > > > pursuing the idea of a metrics persistence system with
> > a
> > > > unique
> > > > > > > > > schema
> > > > > > > > > > > > and possibly a separate datasource, a process
> > initiated by
> > > > a
> > > > > > > > recently
> > > > > > > > > > > > merged PR [1]. Is that still the case? Furthermore,
> > we'd
> > > > need
> > > > > > to
> > > > > > > > > > > > implement data retention and purging, including for the
> > > > current
> > > > > > > > > events
> > > > > > > > > > > > table [2].
> > > > > > > > > > > >
> > > > > > > > > > > > 3) If we consider the events table for metrics
> > storage, we
> > > > must
> > > > > > > > > > > > evaluate average payload sizes. Although a PR [3] was
> > > > > > introduced to
> > > > > > > > > > > > prune large payloads (such as table metadata), this
> > > > > > functionality
> > > > > > > > is
> > > > > > > > > > > > still in its early stages and will evolve. Similar
> > pruning
> > > > > > would be
> > > > > > > > > > > > necessary for metrics reports if they are big.
> > > > > > > > > > > >
> > > > > > > > > > > > 4) As Yong suggested [4], we may still require more
> > > > > > sophisticated
> > > > > > > > > > > > metrics filtering. The events subsystem currently only
> > > > allows
> > > > > > > > > > > > filtering by event type or event category, which may
> > not be
> > > > > > > > granular
> > > > > > > > > > > > enough for our needs (as of today, it would allow only
> > to
> > > > > > > > distinguish
> > > > > > > > > > > > scan vs metrics reports). In that regard, I would
> > welcome
> > > > the
> > > > > > > > > > > > opportunity to implement a generic EventFilter
> > interface
> > > > with a
> > > > > > > > > > > > default implementation based on CEL.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Alex
> > > > > > > > > > > >
> > > > > > > > > > > > [1]: https://github.com/apache/polaris/pull/4397
> > > > > > > > > > > > [2]:
> > > > > > > > >
> > https://lists.apache.org/thread/krmddx8myov926sd0mbh4ogy8sdgrfgq
> > > > > > > > > > > > [3]: https://github.com/apache/polaris/pull/4225
> > > > > > > > > > > > [4]:
> > > > > > > > >
> > https://lists.apache.org/thread/ogskc1szctkg5n0tdj0cm3pfkowcwx4z
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Jun 10, 2026 at 2:04 AM Yufei Gu <
> > > > [email protected]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've been thinking about how Polaris should support
> > > > Iceberg
> > > > > > scan
> > > > > > > > > and
> > > > > > > > > > > > commit
> > > > > > > > > > > > > metrics. A few challenges have come up in recent
> > > > discussions:
> > > > > > > > > > > > > 1. Sync metrics persistence chokes Polaris
> > persistence
> > > > due
> > > > > > to the
> > > > > > > > > high
> > > > > > > > > > > > > volume of scan metrics [3].
> > > > > > > > > > > > > 2. We spent considerable time figuring out the
> > metrics
> > > > > > > > persistence,
> > > > > > > > > > > > > including the schema, SPIs, REST APIs [4].
> > > > > > > > > > > > > 3. Metric filtering remains a challenge [1].
> > > > > > > > > > > > > 4. We need to figure out how to purge metrics because
> > > > they
> > > > > > keep
> > > > > > > > > growing
> > > > > > > > > > > > [2].
> > > > > > > > > > > > >
> > > > > > > > > > > > > Looking at these challenges, most of them are not
> > really
> > > > > > metrics
> > > > > > > > > > > > problems.
> > > > > > > > > > > > > They are transport, delivery, retention, and
> > lifecycle
> > > > > > problems
> > > > > > > > > that
> > > > > > > > > > > the
> > > > > > > > > > > > > existing event framework already addresses. I'd like
> > to
> > > > > > propose
> > > > > > > > > using
> > > > > > > > > > > the
> > > > > > > > > > > > > event system to facilitate the current use cases of
> > > > Iceberg
> > > > > > scan
> > > > > > > > > and
> > > > > > > > > > > > commit
> > > > > > > > > > > > > metrics rather than introducing a separate Polaris
> > > > metrics
> > > > > > > > > subsystem.
> > > > > > > > > > > The
> > > > > > > > > > > > > metrics for current use cases are fundamentally
> > events
> > > > with
> > > > > > > > > structured
> > > > > > > > > > > > > telemetry attached. They are append only, generated
> > by
> > > > IRC
> > > > > > > > > endpoints,
> > > > > > > > > > > > > typically consumed asynchronously, and often
> > forwarded to
> > > > > > > > external
> > > > > > > > > > > > systems.
> > > > > > > > > > > > > Since Polaris already needs to support them as part
> > of
> > > > IRC,
> > > > > > > > > treating
> > > > > > > > > > > them
> > > > > > > > > > > > > as event types seems like a natural fit.
> > > > > > > > > > > > >
> > > > > > > > > > > > > More importantly, I think Polaris should remain a
> > catalog
> > > > > > service
> > > > > > > > > and
> > > > > > > > > > > > > telemetry producer rather than a metrics warehouse.
> > > > Instead
> > > > > > of
> > > > > > > > > > > > introducing
> > > > > > > > > > > > > a dedicated metrics subsystem along with storage,
> > > > retention,
> > > > > > > > > query, and
> > > > > > > > > > > > > scaling concerns, we could build on the existing
> > event
> > > > > > framework:
> > > > > > > > > > > > >
> > > > > > > > > > > > > - Emit them through the existing event mechanism.
> > We
> > > > will
> > > > > > do
> > > > > > > > > that
> > > > > > > > > > > > anyway
> > > > > > > > > > > > > given it's an IRC endpoint.
> > > > > > > > > > > > > - Let custom event listeners route them to the
> > > > > > destination of
> > > > > > > > > > > choice,
> > > > > > > > > > > > > such as Prometheus, Grafana, RDBMSs, or other
> > systems.
> > > > > > > > > > > > > - Reuse the existing event lifecycle, retention,
> > and
> > > > > > delivery
> > > > > > > > > > > models.
> > > > > > > > > > > > If
> > > > > > > > > > > > > temporary persistence is still required, the
> > existing
> > > > > > event
> > > > > > > > > table
> > > > > > > > > > > can
> > > > > > > > > > > > serve
> > > > > > > > > > > > > that purpose. The payload size is manageable given
> > > > that we
> > > > > > > > have
> > > > > > > > > put
> > > > > > > > > > > > the
> > > > > > > > > > > > > loadTable/LoadView response in events.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This approach also gives deployments flexibility to
> > > > filter,
> > > > > > > > > sample, or
> > > > > > > > > > > > > redirect high volume scan metrics without Polaris
> > needing
> > > > > > backend
> > > > > > > > > > > > specific
> > > > > > > > > > > > > metric storage behavior. For example, event
> > listeners can
> > > > > > choose
> > > > > > > > > which
> > > > > > > > > > > > > metric events to process. We don't need to implement
> > > > metric
> > > > > > > > > filtering
> > > > > > > > > > > > logic
> > > > > > > > > > > > > [1].
> > > > > > > > > > > > >
> > > > > > > > > > > > > In short, my proposal is: Events provide the
> > transport
> > > > and
> > > > > > > > > lifecycle
> > > > > > > > > > > > > mechanism, while downstream metrics systems remain
> > > > > > responsible
> > > > > > > > for
> > > > > > > > > > > > storage,
> > > > > > > > > > > > > querying, aggregation, and visualization.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Curious what others think.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1.
> > > > > > > > >
> > https://lists.apache.org/thread/ogskc1szctkg5n0tdj0cm3pfkowcwx4z
> > > > > > > > > > > > > 2.
> > > > > > > > >
> > https://lists.apache.org/thread/5nst0f2ygnl2gj3j910q7m8nk2fvokc7
> > > > > > > > > > > > > 3.
> > > > > > > > >
> > https://lists.apache.org/thread/zp2rvsdkq3mb46722o0hfl0zh7kdqyr8
> > > > > > > > > > > > > 4.
> > > > > > > > >
> > https://lists.apache.org/thread/qj1y7cw4dygcnczmymdwkfkp4ysq41ts
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yufei
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
