(copy from http://stackoverflow.com/questions/35467128/spark-subtractbykey-increases-rdd-cached-memory-size)
I've found a very strange behavior for RDD's (spark 1.6.0 with scala 2.11): When i use subtractByKey on an RDD the resulting RDD should be of equal or smaller size. What i get is an RDD that takes even more space in memory: //Initialize first RDD val rdd1 = sc.parallelize(Array((1,1),(2,2),(3,3))).cache() //dummy action to cache it => size according to webgui: 184 Bytes rdd1.first //Initialize RDD to subtract (empty RDD should result in no change for rdd1) val rdd2 = sc.parallelize(Array[(Int,Int)]()) //perform subtraction val rdd3 = rdd1.subtractByKey(rdd2).cache() //dummy action to cache rdd3 => size according to webgui: 208 Bytes rdd3.first I frist realized this strange behaviour for an RDD of ~200k rows and size 1.3 GB that scaled up to more than 2 GB after subtraction Edit: Tried the example above with more values(10k) => same behaviour. The size increases by ~1.6 times. Also reduceByKey seems to have a similar effect. When i create an RDD by sc.paralellize(rdd3.collect()) the size is the same as for rdd3, so the increased size carries over even if it's extracted from RDD. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/subtractByKey-increases-RDD-size-in-memory-any-ideas-tp26272.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
