I am not sure if it can work on Spark 1.0, but give it a try. or, Maybe you can try: 1) to construct the edges and vertices RDDs respectively with desired storage level. 2) then, to obtain a graph by using Graph(verticesRDD, edgesRDD).
Best, Yifan LI On 28 Oct 2014, at 12:10, Arpit Kumar <arp8...@gmail.com> wrote: > Hi Yifan LI, > I am currently working on Spark 1.0 in which we can't pass edgeStorageLevel > as parameter. It implicitly caches the edges. So I am looking for a > workaround. > > http://spark.apache.org/docs/1.0.0/api/scala/index.html#org.apache.spark.graphx.GraphLoader$ > > Regards, > Arpit > > On Tue, Oct 28, 2014 at 4:25 PM, Yifan LI <iamyifa...@gmail.com> wrote: > Hi Arpit, > > To try this: > > val graph = GraphLoader.edgeListFile(sc, edgesFile, minEdgePartitions = > numPartitions, edgeStorageLevel = StorageLevel.MEMORY_AND_DISK, > vertexStorageLevel = StorageLevel.MEMORY_AND_DISK) > > > Best, > Yifan LI > > > > > On 28 Oct 2014, at 11:17, Arpit Kumar <arp8...@gmail.com> wrote: > >> Any help regarding this issue please? >> >> Regards, >> Arpit >> >> On Sat, Oct 25, 2014 at 8:56 AM, Arpit Kumar <arp8...@gmail.com> wrote: >> Hi all, >> I am using the GrpahLoader class to load graphs from edge list files. But >> then I need to change the storage level of the graph to some other thing >> than MEMORY_ONLY. >> >> val graph = GraphLoader.edgeListFile(sc, fname, >> minEdgePartitions = >> numEPart).persist(StorageLevel.MEMORY_AND_DISK_SER) >> >> The error I am getting while executing this is: >> Exception in thread "main" java.lang.UnsupportedOperationException: Cannot >> change storage level of an RDD after it was already assigned a level >> >> >> Then I looked into the GraphLoader class. I know that in the latest version >> of spark support for setting persistence level is provided in this class. >> Please suggest a workaround for spark 1.0.0 as I do not have the option to >> shift to latest release. >> >> Note: I tried copying the GraphLoader class to my package as GraphLoader1 >> importing >> >> package com.cloudera.xyz >> >> import org.apache.spark.storage.StorageLevel >> import org.apache.spark.graphx._ >> import org.apache.spark.{Logging, SparkContext} >> import org.apache.spark.graphx.impl._ >> >> and then changing the persistence level to my suitability as >> .persist(gStorageLevel) instead of .cache() >> >> But while compiling I am getting the following errors >> >> GraphLoader1.scala:49: error: class EdgePartitionBuilder in package impl >> cannot be accessed in package org.apache.spark.graphx.impl >> [INFO] val builder = new EdgePartitionBuilder[Int, Int] >> >> I am also attaching the file with the mail. Maybe this way of doing thing is >> not possible. >> >> >> Please suggest some workarounds so that I can set persistence level of my >> graph to MEMORY_AND_DISK_SER for the graph I read from edge file list >> >> >> >> -- >> Arpit Kumar >> Fourth Year Undergraduate >> Department of Computer Science and Engineering >> Indian Institute of Technology, Kharagpur > > > > > -- > Arpit Kumar > Fourth Year Undergraduate > Department of Computer Science and Engineering > Indian Institute of Technology, Kharagpur
