unsubscribe

unsubscribe

Re: [EXTERNAL] Use of ML in certain aspects of Spark to improve the performance

unsubscribe From: Mich Talebzadeh Sent: Tuesday, August 8, 2023 4:43 PM To: user @spark Subject: [EXTERNAL] Use of ML in certain aspects of Spark to improve the performance I am currently pondering and sharing my thoughts openly. Given our reliance on gathered

Use of ML in certain aspects of Spark to improve the performance

I am currently pondering and sharing my thoughts openly. Given our reliance on gathered statistics, it prompts the question of whether we could integrate specific machine learning components into Spark Structured Streaming. Consider a scenario where we aim to adjust configuration values on the fly

Re: Dynamic allocation does not deallocate executors

So if you disable shuffle tracking but enable shuffle block decommissioning it should work from memory On Tue, Aug 8, 2023 at 4:13 AM Mich Talebzadeh wrote: > Hm. I don't think it will work > > --conf spark.dynamicAllocation.shuffleTracking.enabled=false > > In Spark 3.4.1 running spark in k8s

Re: Dynamic allocation does not deallocate executors

Hm. I don't think it will work --conf spark.dynamicAllocation.shuffleTracking.enabled=false In Spark 3.4.1 running spark in k8s you get : org.apache.spark.SparkException: Dynamic allocation of executors requires the external shuffle service. You may enable this through