Thanks for the response. From Spark Web-UI's Storage tab, I do see cached RDD there.
But the storage level is "Memory Deserialized 1x Replicated". How can I change the storage level? Because I have a big table there. Thanks! ________________________________ From: Cheng Lian [mailto:lian.cs....@gmail.com] Sent: 2014年9月26日 21:24 To: Haopu Wang; user@spark.apache.org Subject: Re: Spark SQL question: is cached SchemaRDD storage controlled by "spark.storage.memoryFraction"? Yes it is. The in-memory storage used with SchemaRDD also uses RDD.cache() under the hood. On 9/26/14 4:04 PM, Haopu Wang wrote: Hi, I'm querying a big table using Spark SQL. I see very long GC time in some stages. I wonder if I can improve it by tuning the storage parameter. The question is: the schemaRDD has been cached with "cacheTable()" function. So is the cached schemaRDD part of memory storage controlled by the "spark.storage.memoryFraction" parameter? Thanks! --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
