Hi Patrick,
What if all the data has to be keep in cache all time. If applying union
result in new RDD then caching this would result into keeping older as well
as this into memory hence duplicating data.
Below is what i understood from your comment.
sqlContext.cacheTable(existingRDD)// caches the RDD as schema RDD uses
columnar compression
existingRDD.union(newRDD).registerAsTable("newTable")
sqlContext.cacheTable(newTable) -- duplicated data
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Re-Use-Case-of-mutable-RDD-any-ideas-around-will-help-tp14095p14107.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
