Hi Yifan LI, I am currently working on Spark 1.0 in which we can't pass edgeStorageLevel as parameter. It implicitly caches the edges. So I am looking for a workaround.
http://spark.apache.org/docs/1.0.0/api/scala/index.html#org.apache.spark.graphx.GraphLoader$ Regards, Arpit On Tue, Oct 28, 2014 at 4:25 PM, Yifan LI <iamyifa...@gmail.com> wrote: > Hi Arpit, > > To try this: > > val graph = GraphLoader.edgeListFile(sc, edgesFile, minEdgePartitions = > numPartitions, edgeStorageLevel = StorageLevel.MEMORY_AND_DISK, > vertexStorageLevel = StorageLevel.MEMORY_AND_DISK) > > > Best, > Yifan LI > > > > > On 28 Oct 2014, at 11:17, Arpit Kumar <arp8...@gmail.com> wrote: > > Any help regarding this issue please? > > Regards, > Arpit > > On Sat, Oct 25, 2014 at 8:56 AM, Arpit Kumar <arp8...@gmail.com> wrote: > >> Hi all, >> I am using the GrpahLoader class to load graphs from edge list files. But >> then I need to change the storage level of the graph to some other thing >> than MEMORY_ONLY. >> >> val graph = GraphLoader.edgeListFile(sc, fname, >> minEdgePartitions = >> numEPart).persist(StorageLevel.MEMORY_AND_DISK_SER) >> >> The error I am getting while executing this is: >> Exception in thread "main" java.lang.UnsupportedOperationException: >> Cannot change storage level of an RDD after it was already assigned a level >> >> >> Then I looked into the GraphLoader class. I know that in the latest >> version of spark support for setting persistence level is provided in this >> class. Please suggest a workaround for spark 1.0.0 as I do not have the >> option to shift to latest release. >> >> Note: I tried copying the GraphLoader class to my package as GraphLoader1 >> importing >> >> package com.cloudera.xyz >> >> import org.apache.spark.storage.StorageLevel >> import org.apache.spark.graphx._ >> import org.apache.spark.{Logging, SparkContext} >> import org.apache.spark.graphx.impl._ >> >> and then changing the persistence level to my suitability as >> .persist(gStorageLevel) instead of .cache() >> >> But while compiling I am getting the following errors >> >> GraphLoader1.scala:49: error: class EdgePartitionBuilder in package impl >> cannot be accessed in package org.apache.spark.graphx.impl >> [INFO] val builder = new EdgePartitionBuilder[Int, Int] >> >> I am also attaching the file with the mail. Maybe this way of doing thing >> is not possible. >> >> >> Please suggest some workarounds so that I can set persistence level of my >> graph to MEMORY_AND_DISK_SER for the graph I read from edge file list >> > > > > -- > Arpit Kumar > Fourth Year Undergraduate > Department of Computer Science and Engineering > Indian Institute of Technology, Kharagpur > > > -- Arpit Kumar Fourth Year Undergraduate Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur
