Re: Graphframes pattern causing java heap space errors

Thanks Ted for the input. I was able to get it working with pyspark shell but the same job submitted via 'spark-submit' using client or cluster deploy mode ends up with these errors: ~ java.lang.OutOfMemoryError: Java heap space at java.lang.Object.clone(Native Method) at

Re: Graphframes pattern causing java heap space errors

Looks like the exception occurred on driver. Consider increasing the values for the following config: conf.set("spark.driver.memory", "10240m") conf.set("spark.driver.maxResultSize", "2g") Cheers On Sat, Apr 9, 2016 at 9:02 PM, Buntu Dev wrote: > I'm running it via

Re: Graphframes pattern causing java heap space errors

I'm running it via pyspark against yarn in client deploy mode. I do notice in the spark web ui under Environment tab all the options I've set, so I'm guessing these are accepted. On Sat, Apr 9, 2016 at 5:52 PM, Jacek Laskowski wrote: > Hi, > > (I haven't played with

Re: Graphframes pattern causing java heap space errors

Hi, (I haven't played with GraphFrames) What's your `sc.master`? How do you run your application -- spark-submit or java -jar or sbt run or...? The reason I'm asking is that few options might not be in use whatsoever, e.g. spark.driver.memory and spark.executor.memory in local mode. Pozdrawiam,

Graphframes pattern causing java heap space errors

I'm running this motif pattern against 1.5M vertices (5.5mb) and 10M (60mb) edges: tgraph.find("(a)-[]->(b); (c)-[]->(b); (c)-[]->(d)") I keep running into Java heap space errors: ~ ERROR actor.ActorSystemImpl: Uncaught fatal error from thread