obelix74 opened a new pull request, #3385: URL: https://github.com/apache/polaris/pull/3385
<!-- ๐ Describe what changes you're proposing, especially breaking or user-facing changes. ๐ See https://github.com/apache/polaris/blob/main/CONTRIBUTING.md for more. --> ## Checklist - [x] ๐ก๏ธ Don't disclose security issues! (contact [email protected]) - [x] ๐ Clearly explained why the changes are needed, or linked related issues: Fixes # - [x] ๐งช Added/updated tests with good coverage, or manually tested (and explained how) - [x] ๐ก Added comments for complex logic - [ ] ๐งพ Updated `CHANGELOG.md` (if needed) - [x] ๐ Updated documentation in `site/content/in-dev/unreleased` (if needed) ## Overview Implements comprehensive metrics persistence for Iceberg table operations with support for both unified audit trail and analytics-optimized storage strategies. Enables tracking and analysis of scan and commit metrics from compute engines (Spark, Trino, Flink) with full OpenTelemetry trace context integration. Fixes #3337 ## Key Features ### Metrics Persistence - **Dual Storage Strategy**: Supports persisting metrics to both events table (audit trail) and dedicated tables (analytics) - **Flexible Reporting**: Four reporter implementations (default, events, persistence, composite) - **Schema v4**: New dedicated tables for scan and commit metrics with proper indexing - **OpenTelemetry Integration**: Full trace context propagation for correlation across operations - **Automated Cleanup**: Configurable retention policies with scheduled cleanup service ## Implementation Details ### New Components - `PersistingMetricsReporter`: Persists to dedicated scan/commit metrics tables - `EventsMetricsReporter`: Persists to events table as JSON - `CompositeMetricsReporter`: Multi-destination reporting - `MetricsReportCleanupService`: Automated retention management - `MetricsReportingConfiguration`: Flexible configuration interface ### Database Schema - `scan_metrics_report`: 30+ columns for scan metrics with indexes on timestamp, table, trace_id - `commit_metrics_report`: 30+ columns for commit metrics with indexes on timestamp, table, operation - Schema version upgraded from v3 to v4 ### JDBC Persistence Layer - Write methods: `writeScanMetricsReport()`, `writeCommitMetricsReport()` - Query methods: Filter by table, time range, trace ID - Cleanup methods: `deleteAllMetricsReportsOlderThan()` ### Event System Integration - `AfterReportMetricsEvent`: Emitted after metrics reports are processed - `PolarisPersistenceEventListener`: Extracts and persists metrics data - `InMemoryBufferEventListener`: Batch writes for performance ## Configuration polaris: iceberg-metrics: reporting: type: composite # or: default, events, persistence targets: โข events โข persistence retention: enabled: true retention-period: P30D cleanup-interval: PT6H ## Testing - 933 new test lines across unit and integration tests - All unit tests passing - Comprehensive coverage for reporters, persistence, and model classes ## Breaking Changes None. All changes are additive with backward compatibility maintained. ## Documentation - Added telemetry documentation in `site/content/in-dev/unreleased/telemetry.md` - Configuration examples for all reporter types - Query examples for common analytics use cases ## Code breakdown | Category | Added Lines | % of Total | Net Lines | Files | |----------|-------------|------------|-----------|-------| | Core Code | 1,683 | 33.6% | 1,675 | 19 | | Tests | 2,025 | 40.4% | 2,017 | 11 | | Boilerplate | 753 | 15.0% | 753 | 4 | | Schema/SQL | 360 | 7.2% | 360 | 2 | | Documentation | 179 | 3.6% | 179 | 1 | | Build Config | 8 | 0.2% | 7 | 2 | | TOTAL | 5,008 | 100% | 4,991 | 39 | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
