RE: Re: Spark Compatibility with Spring Boot 3.x

Team, Do we have any updates when spark 4.x version will release in order to address below issues related to > java.lang.NoClassDefFoundError: javax/servlet/Servlet Thanks and Regards, Guru On 2023/10/05 17:19:51 Angshuman Bhattacharya wrote: > Thanks Ahmed. I am trying to bring this up with Spa

Re: Tuning Best Practices

Thanks, Jack! Please let me know if you find any other guides specific to tuning shuffles and joins. Currently, the best way I know how to handle joins across large datasets that can't be broadcast is by rewriting the source tables HIVE partitioned by one or two join keys, and then breaking down

[sql] how to connect query stage to Spark job/stages?

Hi, I am seeking advice on measuring the performance of each QueryStage (QS) when AQE is enabled in Spark SQL. Specifically, I need help to automatically map a QS to its corresponding jobs (or stages) to get the QS runtime metrics. I recorded the QS structure via a customized injected Query Sta