1fanwang opened a new issue, #53898: URL: https://github.com/apache/airflow/issues/53898
### Description This feature request proposes adding comprehensive client-side database operation metrics to Airflow to provide detailed visibility into database performance, query patterns, and connection behavior. This would enable better monitoring, debugging, and optimization of Airflow deployments, especially during stress testing and database infrastructure changes. Currently, Airflow only provides basic debug-level SQLAlchemy logging, which is insufficient for production monitoring and performance analysis. This feature would add production-ready metrics that integrate with existing Airflow metrics infrastructure (StatsD, OpenTelemetry). ### Use case/motivation Primary Use Cases: 1. Stress Testing & Load Analysis: During load testing, teams need detailed metrics to understand database bottlenecks, query patterns, and connection pool utilization without impacting production performance. 2. Database Infrastructure Changes: When migrating to new database configurations (e.g., ProxySQL, read replicas, sharding), teams need granular metrics to validate routing rules, measure latency differences, and ensure error rates remain acceptable. 3. Production Monitoring: Continuous monitoring of database operation health, including slow query detection, error rate tracking, and connection pool exhaustion prevention. 4. Performance Optimization: Identifying specific tables, operations, or code paths that generate excessive database load or slow queries. Example Metrics Needed: **Operation Metrics:** Query latency (by operation type: SELECT, INSERT, UPDATE, DELETE) Error rates and error types Query counts by operation type Slow query detection and tracking **Connection Pool Metrics:** Pool size, checked-out connections, available connections Connection creation/checkout/checkin rates Pool overflow usage **Granular Tracking:** Metrics by database table Metrics by calling module/function (for identifying hot paths) Custom operation tagging for business logic ### Related issues This addresses the lack of production-ready database observability in Airflow performance monitoring. While Airflow has extensive task-level metrics, database-level observability has been limited to debug logging. ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
