On Wed, Sep 10, 2014 at 1:05 AM, Boxian Dong <box...@indoo.rs> wrote: > Thank you very much for your kindly help. I rise some another questions: > > - If the RDD is stored in serialized format, is that means that whenever > the RDD is processed, it will be unpacked and packed again from and back to > the JVM even they are located on the same machine? > <http://apache-spark-user-list.1001560.n3.nabble.com/file/n13862/rdd_img.png>
In PySpark, Yes. But in Spark generally, no, you have several choice to cache RDD in Scala, serialized or not. > - Can the RDD be partially unpacked from the serialized state? or when > every a RDD is touched, it must be fully unpacked, and of course pack again > afterword. The items in RDD are deserialized batch by batch, so if you call rdd.take(), only first small parts of items are deserialized. The cache of RDD are kept in JVM, you do not need to pack again after visiting them. > - When a RDD is cached, is it saved in a unserialized format or > serialized format? If it's saved in a unserialized format, is the partially > reading of RDD from JVM to PYTHON runtime possible? For PySpark, they are all saved in serialized format. During a transformation of RDD, you can only see the current partition, you can not access other partitions or other RDD. The RDD always are read-only, so you can not modify them any time. (all the modification will be dropped.) > Thank you very much > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/RDD-memory-questions-tp13805p13862.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
