Hi Prithvi, These are good questions to discuss!
> 1. Confirm that external listeners are the supported production audit path, > and that polaris_schema.events is not. I tend to agree with that statement, with some caveats. I do not think "production" is something Polaris as an ASF project can support in principle. I believe the preferred term is "users". In that regard, I believe the listener SPI is a stronger contract with users than the RDBMS schema of the "events" table. Users who need strong auditing in "production" can use the SPI to direct the events to the appropriate storage system. This way "production" guarantees are controlled by the Polaris user according to local requirements. Users who choose to use the existing RDBMS storage for events are free to do so, of course, but the "production" decision about the suitability of this storage type still rests with the user, not the project. > 2. Whether authentication failures and authorization allow/deny should > become first-class events. Success-only coverage is not enough for an audit > trail. I think this is a good feature to add to Polaris (separately for Auth N/Z). > 3. Whether Polaris should document a production profile that does not > silently drop events, or fail closed when the sink is down. Guaranteed delivery is not an easy feature to implement, if that's what you mean. Ultimately, I think it connects to how changes requested via APIs are committed [1]. Right now, event submission is detached from Persistence commits, AFAIK. A successful commit does not guarantee event delivery, and a failed event submission does not roll back the change that induced it. > 4. Whether phase 1 is docs only [...] Improving docs is always welcome. With the above caveats in mind we can certainly document what users can configure with the current OSS codebase and what runtime guarantees are provided (weak, IMHO). As a side note, I'd also like to mention that NoSQL Persistence offers full change history for Polaris entities. This includes changes to properties, metadata / snapshot locations made during table updates, etc. This data is currently not exposed via any user-facing APIs, but it can be exposed if there's interest. [1] https://lists.apache.org/thread/622d0sn4wzhw993dr5yhmz5zhy4228hp Cheers, Dmitri. On Sun, Sep 20, 2026 at 3:23 PM Prithvi S <[email protected]> wrote: > Hi all, > > I opened https://github.com/apache/polaris/issues/5561 to discuss a > supported audit trail for Polaris operators. > > The event pipeline is already the right foundation: listeners since 1.0, > JDBC events + CloudWatch in 1.2.0 (preview), multi-listener in 1.5.0, the > async executor in 1.6.0, and Kafka plus OpenTelemetry listeners in 1.7.0. > Events can carry principal, event type, timestamp, request id, and > catalog/namespace/table identifiers, and DefaultEventSanitizer keeps > credentials out of the default payload. > > That is not yet an operator-facing audit story. HTTP access logs and the > default server log format do not reliably include the principal. Listener > delivery is best-effort (bounded backlog drops; the JDBC buffer is > in-memory, async, and outside the request transaction). Authentication > failures never reach the event delegator. Authorization denials are INFO > log lines, not PolarisEventTypes. polaris_schema.events has no query API, > secondary indexes, purge, or Polaris ACL, and NoSQL does not implement > writeEvents. > > This is not a request for Polaris to become a SIEM, add a Management API, > or build an in-process query engine. > > Questions for this thread: > > 1. Confirm that external listeners are the supported production audit path, > and that polaris_schema.events is not. > 2. Whether authentication failures and authorization allow/deny should > become first-class events. Success-only coverage is not enough for an audit > trail. > 3. Whether Polaris should document a production profile that does not > silently drop events, or fail closed when the sink is down. > 4. Whether phase 1 is docs only (recommended listener set, delivery > semantics, how to investigate a 403 in the sink), with coverage and > delivery changes as follow-ups. > > Cheers, > Prithvi S >
