StreamingQueryListener integration with Spark native metric sink (JmxSink)

2024-04-18 Thread Mason Chen
Hi all, Is it possible to integrate StreamingQueryListener with Spark metrics so that metrics can be reported through Spark's internal metric system? Ideally, I would like to report some custom metrics through StreamingQueryListener and export them to Spark's JmxSink. Best, Mason

[ANNOUNCE] Apache Spark 3.4.3 released

2024-04-18 Thread Dongjoon Hyun
We are happy to announce the availability of Apache Spark 3.4.3! Spark 3.4.3 is a maintenance release containing many fixes including security and correctness domains. This release is based on the branch-3.4 maintenance branch of Spark. We strongly recommend all 3.4 users to upgrade to this