Re: Question About OFF_HEAP Caching
Here is one blog illustrating how to use Spark on Alluxio for this purpose. Hope it will help: http://www.alluxio.com/2016/04/getting-started-with-alluxio-and-spark/ On Mon, Jul 18, 2016 at 6:36 AM, Gene Pang wrote: > Hi, > > If you want to use Alluxio with Spark 2.x, it is recommended to write to > and read from Alluxio with files. You can save an RDD with saveAsObjectFile > with an Alluxio path (alluxio://host:port/path/to/file), and you can read > that file from any other Spark job. Here is additional information on how > to run Spark with Alluxio: > http://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html > > Hope that helps, > Gene > > On Mon, Jul 18, 2016 at 12:11 AM, condor join > wrote: > >> Hi All, >> >> I have some questions about OFF_HEAP Caching. In Spark 1.X when we use >> *rdd.persist(StorageLevel.OFF_HEAP)*,that means rdd caching in >> Tachyon(Alluxio). However,in Spark 2.X,we can directly use OFF_HEAP For >> Caching >> >> ( >> https://issues.apache.org/jira/browse/SPARK-13992?jql=project%20%3D%20SPARK%20AND%20text%20~%20%22off-heap%20caching%22). >> I am confuse about this and I have follow questions: >> >> 1.In Spark 2.X, how should we use Tachyon for caching? >> >> 2.Is there any reason that must change in this way(I mean use off_heap >> directly instead of using Tachyon) >> >> Thanks a lot! >> >> >> >> >
Re: Question About OFF_HEAP Caching
Hi, If you want to use Alluxio with Spark 2.x, it is recommended to write to and read from Alluxio with files. You can save an RDD with saveAsObjectFile with an Alluxio path (alluxio://host:port/path/to/file), and you can read that file from any other Spark job. Here is additional information on how to run Spark with Alluxio: http://www.alluxio.org/docs/master/en/Running-Spark-on-Alluxio.html Hope that helps, Gene On Mon, Jul 18, 2016 at 12:11 AM, condor join wrote: > Hi All, > > I have some questions about OFF_HEAP Caching. In Spark 1.X when we use > *rdd.persist(StorageLevel.OFF_HEAP)*,that means rdd caching in > Tachyon(Alluxio). However,in Spark 2.X,we can directly use OFF_HEAP For > Caching > > ( > https://issues.apache.org/jira/browse/SPARK-13992?jql=project%20%3D%20SPARK%20AND%20text%20~%20%22off-heap%20caching%22). > I am confuse about this and I have follow questions: > > 1.In Spark 2.X, how should we use Tachyon for caching? > > 2.Is there any reason that must change in this way(I mean use off_heap > directly instead of using Tachyon) > > Thanks a lot! > > > >
Question About OFF_HEAP Caching
Hi All, I have some questions about OFF_HEAP Caching. In Spark 1.X when we use rdd.persist(StorageLevel.OFF_HEAP),that means rdd caching in Tachyon(Alluxio). However,in Spark 2.X,we can directly use OFF_HEAP For Caching (https://issues.apache.org/jira/browse/SPARK-13992?jql=project%20%3D%20SPARK%20AND%20text%20~%20%22off-heap%20caching%22). I am confuse about this and I have follow questions: 1.In Spark 2.X, how should we use Tachyon for caching? 2.Is there any reason that must change in this way(I mean use off_heap directly instead of using Tachyon) Thanks a lot!