Hi, I moved my Pyspark to 2.0.1 and now I can connect. However, I cannot execute any job. I always get an "16/12/01 09:37:07 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources" error. I added more resources to the nodes, restricted the default cores and memory so Pyspark does not consume all, but still I cannot count the size of an 5 element RDD!
Any Idea what might be causing this? Best, Klaus From: Felix Cheung [mailto:felixcheun...@hotmail.com] Sent: Thursday, December 1, 2016 12:44 AM To: user@spark.apache.org; Schaefers, Klaus <klaus.schaef...@philips.com> Subject: Re: PySpark to remote cluster Spark 2.0.1 is running with a different py4j library than Spark 1.6. You will probably run into other problems mixing versions though - is there a reason you can't run Spark 1.6 on the client? _____________________________ From: Klaus Schaefers <klaus.schaef...@philips.com<mailto:klaus.schaef...@philips.com>> Sent: Wednesday, November 30, 2016 2:44 AM Subject: PySpark to remote cluster To: <user@spark.apache.org<mailto:user@spark.apache.org>> Hi, I want to connect with a local Jupyter Notebook to a remote Spark cluster. The Cluster is running Spark 2.0.1 and the Jupyter notebook is based on Spark 1.6 and running in a docker image (Link). I try to init the SparkContext like this: import pyspark sc = pyspark.SparkContext('spark://<IP>:7077') However, this gives me the following exception: ERROR:py4j.java_gateway:Error while sending or receiving. Traceback (most recent call last): File "/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 746, in send_command raise Py4JError("Answer from Java side is empty") py4j.protocol.Py4JError: Answer from Java side is empty During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 626, in send_command response = connection.send_command(command) File "/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 750, in send_command raise Py4JNetworkError("Error while sending or receiving", e) py4j.protocol.Py4JNetworkError: Error while sending or receiving During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 740, in send_command answer = smart_decode(self.stream.readline()[:-1]) File "/opt/conda/lib/python3.5/socket.py", line 575, in readinto return self._sock.recv_into(b) ConnectionResetError: [Errno 104] Connection reset by peer ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server Traceback (most recent call last): File "/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 746, in send_command raise Py4JError("Answer from Java side is empty") py4j.protocol.Py4JError: Answer from Java side is empty ... During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/lib/python3.5/site-packages/IPython/utils/PyColorize.py", line 262, in format2 for atoken in generate_tokens(text.readline): File "/opt/conda/lib/python3.5/tokenize.py", line 597, in _tokenize raise TokenError("EOF in multi-line statement", (lnum, 0)) tokenize.TokenError: ('EOF in multi-line statement', (2, 0)) Is this error caused by the different spark versions? Best, Klaus -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/PySpark-to-remote-cluster-tp28147.html Sent from the Apache Spark User List mailing list archive at Nabble.com<http://Nabble.com>. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> ________________________________ The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
