Re: spark keeps on creating executors and each one fails with "TransportClient has not yet been set."

Anyone has any idea what could I enable so as to find out what it is trying to connect to? On Thu, Mar 2, 2017 at 5:34 PM, Aseem Bansal wrote: > Is there a way to find out what is it trying to connect to? I am running > my spark client from within a docker container so I

How to tune groupBy operations in Spark 2.x?

Hi, How to tune the Spark Jobs that use groupBy operations? Earlier I used to use --conf spark.shuffle.memoryFraction=0.8 --conf spark.storage.memoryFraction=0.1 to tune my jobs that use groupBy. But, with Spark 2.x this configs seem to have been deprecated. What would be the appropriate

SimpleConfigObject

Hi, How to read json string from SimpleConfigObject. SimpleConfigObject({"ID":"123","fileName":"123.txt"}) Regards, Rajesh

spark keeps on creating executors and each one fails with "TransportClient has not yet been set."

Is there a way to find out what is it trying to connect to? I am running my spark client from within a docker container so I opened up various ports as per http://stackoverflow.com/questions/27729010/how-to-configure-apache-spark-random-worker-ports-for-tight-firewalls after adding all the

Restart if driver gets insufficient resources

Hi All, We are running spark on kubernetes. There is a scenario in which the spark driver(pod) was not able to communicate properly with master and it got stuck saying insufficient resources. On restarting the spark driver (pod) manually , It was able to run properly. Is there a way to just

strange usage of tempfile.mkdtemp() in PySpark mllib.recommendation doctest

Dear Spark user mailinglist members, In PySpark's mllib.recommendation doctest, I found a bit strange usage of temporary directory creation function, tempfile.mkdtemp(), in the following part. # https://github.com/apache/spark/blob/master/python/pyspark/mllib/recommendation.py ...