Does anybody use spark.rpc.io.mode=epoll?

2017-03-07 Thread Steven Ruppert
The epoll mode definitely exists in spark, but the official documentation does not mention it, nor any of the other settings that appear to be unofficially documented in: https://github.com/jaceklaskowski/mastering-apache-spark-book/blob/master/spark-rpc-netty.adoc I don't seem to have any

Re: spark-shell running out of memory even with 6GB ?

2017-01-09 Thread Steven Ruppert
The spark-shell process alone shouldn't take up that much memory, at least in my experience. Have you dumped the heap to see what's all in there? What environment are you running spark in? Doing stuff like RDD.collect() or .countByKey will pull potentially a lot of data the spark-shell heap.