Re: How to unpersist JavaPairRDD
Thanks Josh On Wed, Nov 20, 2013 at 5:01 PM, Josh Rosen wrote: > JavaPairRDD should have had a unpersist() method; we'll fix this bug in > 0.8.1 (see https://github.com/apache/incubator-spark/pull/103). In the > meantime, just call myJavaPairRDD.rdd().unpersist() (see > https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201310.mbox/%3ccaoepxp5syqn7v9gdomjcvgqzvaad-epewxi025jpmu_neic...@mail.gmail.com%3E > ). > > > On Wed, Nov 20, 2013 at 4:39 PM, sasmita Patra wrote: > >> Hi, >> I have two datasets that i load from HDFS file system. After loading the >> files, i cache the datasets. >> >> I have a requirement to do join on these datasets (LEFT/RIGHT INNER/OUTER >> JOIN) and apply some filter conditions and then run multiple queries on >> this joined filtered dataset. >> >> I have created JavaPairRDDs for the datasets, doing the join, applying >> filter and caching the dataset. >> >> After running the queries, i need to unpersist this dataset. Currently i >> don't see any unpersist api for JavaPairRDD. >> >> Is there any way to unpersist the JavaPairRDD? >> >> Thanks, >> Sasmita >> >> >
Re: How to unpersist JavaPairRDD
JavaPairRDD should have had a unpersist() method; we'll fix this bug in 0.8.1 (see https://github.com/apache/incubator-spark/pull/103). In the meantime, just call myJavaPairRDD.rdd().unpersist() (see https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201310.mbox/%3ccaoepxp5syqn7v9gdomjcvgqzvaad-epewxi025jpmu_neic...@mail.gmail.com%3E ). On Wed, Nov 20, 2013 at 4:39 PM, sasmita Patra wrote: > Hi, > I have two datasets that i load from HDFS file system. After loading the > files, i cache the datasets. > > I have a requirement to do join on these datasets (LEFT/RIGHT INNER/OUTER > JOIN) and apply some filter conditions and then run multiple queries on > this joined filtered dataset. > > I have created JavaPairRDDs for the datasets, doing the join, applying > filter and caching the dataset. > > After running the queries, i need to unpersist this dataset. Currently i > don't see any unpersist api for JavaPairRDD. > > Is there any way to unpersist the JavaPairRDD? > > Thanks, > Sasmita > >
How to unpersist JavaPairRDD
Hi, I have two datasets that i load from HDFS file system. After loading the files, i cache the datasets. I have a requirement to do join on these datasets (LEFT/RIGHT INNER/OUTER JOIN) and apply some filter conditions and then run multiple queries on this joined filtered dataset. I have created JavaPairRDDs for the datasets, doing the join, applying filter and caching the dataset. After running the queries, i need to unpersist this dataset. Currently i don't see any unpersist api for JavaPairRDD. Is there any way to unpersist the JavaPairRDD? Thanks, Sasmita
