The pyspark app stdout/err log shows this oddity.
Traceback (most recent call last):
File "/root/spark/notebooks/ingest/XXX.py", line 86, in
print pdfRDD.collect()[:5]
File "/root/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 773,
in collect
File
"/root/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line
536, in __call__
File
"/root/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line
364, in send_command
File
"/root/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line
473, in send_command
File "/usr/lib64/python2.7/socket.py", line 430, in readline
data = recv(1)
KeyboardInterrupt
On 12/02/2015 08:57 PM, Jim Lohse wrote:
Is this the stderr output from a woker? Are any files being written?
Can you run in debug and see how far it's getting?
This to me doesn't give me a direction to look without the actual logs
from $SPARK_HOME or the stderr from the worker UI.
Just imho maybe someone know what this means but it seems like it
could be caused by a lot of things.
On 12/2/2015 6:48 PM, Darren Govoni wrote:
Hi all,
Wondering if someone can provide some insight why this pyspark app
is just hanging. Here is output.
...
15/12/03 01:47:05 INFO TaskSetManager: Starting task 21.0 in stage
0.0 (TID 21, 10.65.143.174, PROCESS_LOCAL, 1794787 bytes)
15/12/03 01:47:05 INFO TaskSetManager: Starting task 22.0 in stage
0.0 (TID 22, 10.97.144.52, PROCESS_LOCAL, 1801814 bytes)
15/12/03 01:47:05 INFO TaskSetManager: Starting task 23.0 in stage
0.0 (TID 23, 10.65.67.146, PROCESS_LOCAL, 1823921 bytes)
15/12/03 01:47:05 INFO TaskSetManager: Starting task 24.0 in stage
0.0 (TID 24, 10.144.176.22, PROCESS_LOCAL, 1820713 bytes)
15/12/03 01:47:05 INFO TaskSetManager: Starting task 25.0 in stage
0.0 (TID 25, 10.65.143.174, PROCESS_LOCAL, 1850492 bytes)
15/12/03 01:47:05 INFO TaskSetManager: Starting task 26.0 in stage
0.0 (TID 26, 10.97.144.52, PROCESS_LOCAL, 1845557 bytes)
15/12/03 01:47:05 INFO TaskSetManager: Starting task 27.0 in stage
0.0 (TID 27, 10.65.67.146, PROCESS_LOCAL, 1876187 bytes)
15/12/03 01:47:05 INFO TaskSetManager: Starting task 28.0 in stage
0.0 (TID 28, 10.144.176.22, PROCESS_LOCAL, 2054748 bytes)
15/12/03 01:47:05 INFO TaskSetManager: Starting task 29.0 in stage
0.0 (TID 29, 10.65.143.174, PROCESS_LOCAL, 1967659 bytes)
15/12/03 01:47:05 INFO TaskSetManager: Starting task 30.0 in stage
0.0 (TID 30, 10.97.144.52, PROCESS_LOCAL, 1977909 bytes)
15/12/03 01:47:05 INFO TaskSetManager: Starting task 31.0 in stage
0.0 (TID 31, 10.65.67.146, PROCESS_LOCAL, 2084044 bytes)
15/12/03 01:47:06 INFO BlockManagerInfo: Added broadcast_0_piece0 in
memory on 10.65.143.174:39356 (size: 5.2 KB, free: 4.1 GB)
15/12/03 01:47:06 INFO BlockManagerInfo: Added broadcast_0_piece0 in
memory on 10.144.176.22:40904 (size: 5.2 KB, free: 4.1 GB)
15/12/03 01:47:06 INFO BlockManagerInfo: Added broadcast_0_piece0 in
memory on 10.97.144.52:35646 (size: 5.2 KB, free: 4.1 GB)
15/12/03 01:47:06 INFO BlockManagerInfo: Added broadcast_0_piece0 in
memory on 10.65.67.146:44110 (size: 5.2 KB, free: 4.1 GB)
...
In the spark console, it says 0/32 tasks and just sits there. No
movement.
Thanks in advance,
D
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
