Re: Exact meaning of spark.memory.storageFraction in spark 2.3.x [Marketing Mail] [Marketing Mail]

Hi Iacovos, thansk for the reply its super clear. Do you know if there is a way to know the max memory usage? In the spark ui 2.3.x the "peak memory usage" metris is always at zero. Thanks, Michel

Re: Exact meaning of spark.memory.storageFraction in spark 2.3.x [Marketing Mail] [Marketing Mail]

This is just a counter to show you the size of cached RDDs. If it is zero means that no caching has occurred. Also, even storage memory is used for computing the counter will show as zero. Iacovos On 20/3/20 4:51 μ.μ., Michel Sumbul wrote: Hi, Thanks for the very quick reply! If I see the me