[
https://issues.apache.org/jira/browse/SPARK-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041517#comment-14041517
]
Matthew Farrellee commented on SPARK-2244:
------------------------------------------
notes -
import logging
logger = logging.getLogger('py4j')
logger.setLevel(logging.DEBUG)
sh = logging.StreamHandler()
logger.addHandler(sh)
one = sc.parallelize(range(1))
one.count()
[repeat until hang]
***successful count -***
>>> one.count()
Command to send: c
o6
setCallSite
scount at <stdin>:1
e
Answer received: yv
Command to send: i
java.util.ArrayList
e
Answer received: ylo150
Command to send: c
o14
classTag
e
Answer received: yro151
Command to send: m
d
o85
e
Answer received: yv
Command to send: m
d
o86
e
Answer received: yv
Command to send: m
d
o87
e
Answer received: yv
Command to send: m
d
o88
e
Answer received: yv
Command to send: m
d
o89
e
Answer received: yv
Command to send: m
d
o90
e
Answer received: yv
Command to send: m
d
o91
e
Answer received: yv
Command to send: m
d
o92
e
Answer received: yv
Command to send: m
d
o93
e
Answer received: yv
Command to send: m
d
o94
e
Answer received: yv
Command to send: m
d
o95
e
Answer received: yv
Command to send: m
d
o96
e
Answer received: yv
Command to send: m
d
o97
e
Answer received: yv
Command to send: m
d
o98
e
Answer received: yv
Command to send: m
d
o99
e
Answer received: yv
Command to send: m
d
o100
e
Answer received: yv
Command to send: m
d
o101
e
Answer received: yv
Command to send: m
d
o102
e
Answer received: yv
Command to send: m
d
o103
e
Answer received: yv
Command to send: m
d
o104
e
Answer received: yv
Command to send: m
d
o105
e
Answer received: yv
Command to send: m
d
o106
e
Answer received: yv
Command to send: m
d
o107
e
Answer received: yv
Command to send: m
d
o108
e
Answer received: yv
Command to send: m
d
o109
e
Answer received: yv
Command to send: m
d
o110
e
Answer received: yv
Command to send: m
d
o111
e
Answer received: yv
Command to send: m
d
o112
e
Answer received: yv
Command to send: m
d
o113
e
Answer received: yv
Command to send: m
d
o114
e
Answer received: yv
Command to send: m
d
o115
e
Answer received: yv
Command to send: m
d
o116
e
Answer received: yv
Command to send: m
d
o117
e
Answer received: yv
Command to send: m
d
o118
e
Answer received: yv
Command to send: m
d
o119
e
Answer received: yv
Command to send: m
d
o120
e
Answer received: yv
Command to send: m
d
o121
e
Answer received: yv
Command to send: m
d
o122
e
Answer received: yv
Command to send: m
d
o123
e
Answer received: yv
Command to send: m
d
o124
e
Answer received: yv
Command to send: m
d
o125
e
Answer received: yv
Command to send: m
d
o126
e
Answer received: yv
Command to send: m
d
o127
e
Answer received: yv
Command to send: m
d
o128
e
Answer received: yv
Command to send: m
d
o129
e
Answer received: yv
Command to send: m
d
o130
e
Answer received: yv
Command to send: m
d
o131
e
Answer received: yv
Command to send: m
d
o132
e
Answer received: yv
Command to send: m
d
o133
e
Answer received: yv
Command to send: m
d
o134
e
Answer received: yv
Command to send: m
d
o135
e
Answer received: yv
Command to send: m
d
o136
e
Answer received: yv
Command to send: m
d
o137
e
Answer received: yv
Command to send: m
d
o138
e
Answer received: yv
Command to send: m
d
o139
e
Answer received: yv
Command to send: m
d
o140
e
Answer received: yv
Command to send: m
d
o141
e
Answer received: yv
Command to send: m
d
o142
e
Answer received: yv
Command to send: m
d
o143
e
Answer received: yv
Command to send: m
d
o144
e
Answer received: yv
Command to send: m
d
o145
e
Answer received: yv
Command to send: m
d
o146
e
Answer received: yv
Command to send: m
d
o147
e
Answer received: yv
Command to send: m
d
o148
e
Answer received: yv
Command to send: m
d
o149
e
Answer received: yv
Command to send: i
java.util.HashMap
e
Answer received: yao152
Command to send: i
java.util.ArrayList
e
Answer received: ylo153
Command to send: r
u
PythonRDD
rj
e
Answer received: ycorg.apache.spark.api.python.PythonRDD
Command to send: c
o14
rdd
e
Answer received: yro154
Command to send: i
org.apache.spark.api.python.PythonRDD
ro154
jgAIoY3B5c3BhcmsuY2xvdWRwaWNrbGUKX21vZHVsZXNfdG9fbWFpbgpxAF1xAVULcHlzcGFyay5yZGRxAmGFcQNSMWNweXNwYXJrLmNsb3VkcGlja2xlCl9maWxsX2Z1bmN0aW9uCnEEKGNweXNwYXJrLmNsb3VkcGlja2xlCl9tYWtlX3NrZWxfZnVuYwpxBWNuZXcKY29kZQpxBihLAksCSwVLE1UWiAAAfAAAiAEAfAAAfAEAgwIAgwIAU3EHToVxCClVBXNwbGl0cQlVCGl0ZXJhdG9ycQqGcQtVQi9ob21lL21hdHQvRG9jdW1lbnRzL1JlcG9zaXRvcmllcy9zcGFyay9kaXN0L3B5dGhvbi9weXNwYXJrL3JkZC5weXEMVQ1waXBlbGluZV9mdW5jcQ1N+wVVAgABcQ5VBGZ1bmNxD1UJcHJldl9mdW5jcRCGcREpdHESUnETSwJ9cRSHcRVScRZ9cRdOXXEYKChoAF1xGWgCYYVxGlIxaAQoaAVoBihLAksCSwJLE1UKiAAAfAEAgwEAU3EbToVxHClVAXNxHWgKhnEeVUIvaG9tZS9tYXR0L0RvY3VtZW50cy9SZXBvc2l0b3JpZXMvc3BhcmsvZGlzdC9weXRob24vcHlzcGFyay9yZGQucHlxH2gPTR4BVQBxIFUBZnEhhXEiKXRxI1JxJEsBaBSHcSVScSZ9cSdOXXEoKGgAXXEpaAJhhXEqUjFoBChoBWgGKEsBSwNLBEszVVNkAAB9AQB4MgB8AABEXSoAfQIAfAEAZAAAawgAcigAfAIAfQEAcQ0AiAAAfAIAfAEAgwIAfQEAcQ0AV3wBAGQAAGsJAHJPAHwBAFYBbgAAZAAAU3ErToVxLFUETm9uZXEthXEuaApVA2FjY3EvVQNvYmpxMIdxMVVCL2hvbWUvbWF0dC9Eb2N1bWVudHMvUmVwb3NpdG9yaWVzL3NwYXJrL2Rpc3QvcHl0aG9uL3B5c3BhcmsvcmRkLnB5cTJoD02kAlUOAAEGAQ0BDAEJAhMBDAFxM2ghhXE0KXRxNVJxNksBaBSHcTdScTh9cTlOXXE6Y29wZXJhdG9yCmFkZApxO2F9cTx0UmF9cT10UihoAF1xPmgCYYVxP1IxaAQoaAVoE0sCaBSHcUBScUF9cUJOXXFDKChoAF1xRGgCYYVxRVIxaAQoaAVoJEsBaBSHcUZScUd9cUhOXXFJaAQoaAVoBihLAUsBSwJLU1UNdAAAfAAAgwEAZwEAU3FKToVxS1UDc3VtcUyFcU1VAXhxToVxT1VCL2hvbWUvbWF0dC9Eb2N1bWVudHMvUmVwb3NpdG9yaWVzL3NwYXJrL2Rpc3QvcHl0aG9uL3B5c3BhcmsvcmRkLnB5cVBVCDxsYW1iZGE+cVFN/QJoICkpdHFSUnFTSwBoFIdxVFJxVX1xVk5dcVd9cVh0UmF9cVl0UihoAF1xWmgCYYVxW1IxaAQoaAVoJEsBaBSHcVxScV19cV5OXXFfaAQoaAVoBihLAUsBSwNLU1UXdAAAZAEAhAAAfAAARIMBAIMBAGcBAFNxYE5oBihLAUsCSwJLc1UVfAAAXQsAfQEAZAAAVgFxAwBkAQBTcWFLAU6GcWIpVQIuMHFjVQFfcWSGcWVVQi9ob21lL21hdHQvRG9jdW1lbnRzL1JlcG9zaXRvcmllcy9zcGFyay9kaXN0L3B5dGhvbi9weXNwYXJrL3JkZC5weXFmVQk8Z2VuZXhwcj5xZ00GA1UCBgBxaCkpdHFpUnFqhnFraEyFcWxVAWlxbYVxblVCL2hvbWUvbWF0dC9Eb2N1bWVudHMvUmVwb3NpdG9yaWVzL3NwYXJrL2Rpc3QvcHl0aG9uL3B5c3BhcmsvcmRkLnB5cW9oUU0GA2ggKSl0cXBScXFLAGgUh3FyUnFzfXF0Tl1xdX1xdnRSYX1xd3RSZX1xeHRSZX1xeXRSY3B5c3Bhcmsuc2VyaWFsaXplcnMKUGlja2xlU2VyaWFsaXplcgpxeimBcXt9cXxVE19vbmx5X3dyaXRlX3N0cmluZ3NxfYlzYmNweXNwYXJrLnNlcmlhbGl6ZXJzCkJhdGNoZWRTZXJpYWxpemVyCnF+KYFxf31xgChVCWJhdGNoU2l6ZXGBTQAEVQpzZXJpYWxpemVycYJoe3Vih3GDLg==
ro152
ro153
bFalse
spython
ro150
ro9
ro151
e
Answer received: yro155
Command to send: c
o155
asJavaRDD
e
Answer received: yro156
Command to send: c
o156
collect
e
Answer received: ylo157
Command to send: c
o157
iterator
e
Answer received: ygo158
Command to send: c
o6
setCallSite
n
e
Answer received: yv
Command to send: c
z:org.apache.spark.api.python.PythonRDD
writeToFile
ro158
s/tmp/spark-f48efc58-280f-4483-8640-e5f6fef530f5/tmp202ulD
e
Answer received: yv
1
***hung count -***
>>> one.count()
Command to send: c
o6
setCallSite
scount at <stdin>:1
e
Answer received: yv
Command to send: i
java.util.ArrayList
e
Answer received: ylo159
Command to send: c
o14
classTag
e
Answer received: yro160
Command to send: i
java.util.HashMap
e
Answer received: yao161
Command to send: i
java.util.ArrayList
e
Answer received: ylo162
Command to send: r
u
PythonRDD
rj
e
Answer received: ycorg.apache.spark.api.python.PythonRDD
Command to send: c
o14
rdd
e
Answer received: yro163
Command to send: i
org.apache.spark.api.python.PythonRDD
ro163
jgAIoY3B5c3BhcmsuY2xvdWRwaWNrbGUKX21vZHVsZXNfdG9fbWFpbgpxAF1xAVULcHlzcGFyay5yZGRxAmGFcQNSMWNweXNwYXJrLmNsb3VkcGlja2xlCl9maWxsX2Z1bmN0aW9uCnEEKGNweXNwYXJrLmNsb3VkcGlja2xlCl9tYWtlX3NrZWxfZnVuYwpxBWNuZXcKY29kZQpxBihLAksCSwVLE1UWiAAAfAAAiAEAfAAAfAEAgwIAgwIAU3EHToVxCClVBXNwbGl0cQlVCGl0ZXJhdG9ycQqGcQtVQi9ob21lL21hdHQvRG9jdW1lbnRzL1JlcG9zaXRvcmllcy9zcGFyay9kaXN0L3B5dGhvbi9weXNwYXJrL3JkZC5weXEMVQ1waXBlbGluZV9mdW5jcQ1N+wVVAgABcQ5VBGZ1bmNxD1UJcHJldl9mdW5jcRCGcREpdHESUnETSwJ9cRSHcRVScRZ9cRdOXXEYKChoAF1xGWgCYYVxGlIxaAQoaAVoBihLAksCSwJLE1UKiAAAfAEAgwEAU3EbToVxHClVAXNxHWgKhnEeVUIvaG9tZS9tYXR0L0RvY3VtZW50cy9SZXBvc2l0b3JpZXMvc3BhcmsvZGlzdC9weXRob24vcHlzcGFyay9yZGQucHlxH2gPTR4BVQBxIFUBZnEhhXEiKXRxI1JxJEsBaBSHcSVScSZ9cSdOXXEoKGgAXXEpaAJhhXEqUjFoBChoBWgGKEsBSwNLBEszVVNkAAB9AQB4MgB8AABEXSoAfQIAfAEAZAAAawgAcigAfAIAfQEAcQ0AiAAAfAIAfAEAgwIAfQEAcQ0AV3wBAGQAAGsJAHJPAHwBAFYBbgAAZAAAU3ErToVxLFUETm9uZXEthXEuaApVA2FjY3EvVQNvYmpxMIdxMVVCL2hvbWUvbWF0dC9Eb2N1bWVudHMvUmVwb3NpdG9yaWVzL3NwYXJrL2Rpc3QvcHl0aG9uL3B5c3BhcmsvcmRkLnB5cTJoD02kAlUOAAEGAQ0BDAEJAhMBDAFxM2ghhXE0KXRxNVJxNksBaBSHcTdScTh9cTlOXXE6Y29wZXJhdG9yCmFkZApxO2F9cTx0UmF9cT10UihoAF1xPmgCYYVxP1IxaAQoaAVoE0sCaBSHcUBScUF9cUJOXXFDKChoAF1xRGgCYYVxRVIxaAQoaAVoJEsBaBSHcUZScUd9cUhOXXFJaAQoaAVoBihLAUsBSwJLU1UNdAAAfAAAgwEAZwEAU3FKToVxS1UDc3VtcUyFcU1VAXhxToVxT1VCL2hvbWUvbWF0dC9Eb2N1bWVudHMvUmVwb3NpdG9yaWVzL3NwYXJrL2Rpc3QvcHl0aG9uL3B5c3BhcmsvcmRkLnB5cVBVCDxsYW1iZGE+cVFN/QJoICkpdHFSUnFTSwBoFIdxVFJxVX1xVk5dcVd9cVh0UmF9cVl0UihoAF1xWmgCYYVxW1IxaAQoaAVoJEsBaBSHcVxScV19cV5OXXFfaAQoaAVoBihLAUsBSwNLU1UXdAAAZAEAhAAAfAAARIMBAIMBAGcBAFNxYE5oBihLAUsCSwJLc1UVfAAAXQsAfQEAZAAAVgFxAwBkAQBTcWFLAU6GcWIpVQIuMHFjVQFfcWSGcWVVQi9ob21lL21hdHQvRG9jdW1lbnRzL1JlcG9zaXRvcmllcy9zcGFyay9kaXN0L3B5dGhvbi9weXNwYXJrL3JkZC5weXFmVQk8Z2VuZXhwcj5xZ00GA1UCBgBxaCkpdHFpUnFqhnFraEyFcWxVAWlxbYVxblVCL2hvbWUvbWF0dC9Eb2N1bWVudHMvUmVwb3NpdG9yaWVzL3NwYXJrL2Rpc3QvcHl0aG9uL3B5c3BhcmsvcmRkLnB5cW9oUU0GA2ggKSl0cXBScXFLAGgUh3FyUnFzfXF0Tl1xdX1xdnRSYX1xd3RSZX1xeHRSZX1xeXRSY3B5c3Bhcmsuc2VyaWFsaXplcnMKUGlja2xlU2VyaWFsaXplcgpxeimBcXt9cXxVE19vbmx5X3dyaXRlX3N0cmluZ3NxfYlzYmNweXNwYXJrLnNlcmlhbGl6ZXJzCkJhdGNoZWRTZXJpYWxpemVyCnF+KYFxf31xgChVCWJhdGNoU2l6ZXGBTQAEVQpzZXJpYWxpemVycYJoe3Vih3GDLg==
ro161
ro162
bFalse
spython
ro159
ro9
ro160
e
Answer received: yro164
Command to send: c
o164
asJavaRDD
e
Answer received: yro165
Command to send: c
o165
collect
e
> pyspark - RDD action hangs (after previously succeeding)
> --------------------------------------------------------
>
> Key: SPARK-2244
> URL: https://issues.apache.org/jira/browse/SPARK-2244
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.1.0
> Environment: system: fedora 20 w/ maven 3.1.1 and openjdk 1.7.0_55
> code: sha b88238fa (master on 23 june 2014)
> cluster: make-distribution.sh followed by ./dist/sbin/start-all.sh (running
> locally)
> Reporter: Matthew Farrellee
> Labels: openjdk, pyspark, python, shell, spark
>
> $ ./dist/bin/pyspark
> Python 2.7.5 (default, Feb 19 2014, 13:47:28)
> [GCC 4.8.2 20131212 (Red Hat 4.8.2-7)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /__ / .__/\_,_/_/ /_/\_\ version 1.0.0-SNAPSHOT
> /_/
> Using Python version 2.7.5 (default, Feb 19 2014 13:47:28)
> SparkContext available as sc.
> >>> hundy = sc.parallelize(range(100))
> >>> hundy.count()
> 100
> >>> hundy.count()
> 100
> >>> hundy.count()
> 100
> [repeat until hang, ctrl-C to get]
> >>> hundy.count()
> ^CTraceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/home/matt/Documents/Repositories/spark/dist/python/pyspark/rdd.py",
> line 774, in count
> return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
> File "/home/matt/Documents/Repositories/spark/dist/python/pyspark/rdd.py",
> line 765, in sum
> return self.mapPartitions(lambda x: [sum(x)]).reduce(operator.add)
> File "/home/matt/Documents/Repositories/spark/dist/python/pyspark/rdd.py",
> line 685, in reduce
> vals = self.mapPartitions(func).collect()
> File "/home/matt/Documents/Repositories/spark/dist/python/pyspark/rdd.py",
> line 649, in collect
> bytesInJava = self._jrdd.collect().iterator()
> File
> "/home/matt/Documents/Repositories/spark/dist/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py",
> line 535, in __call__
> File
> "/home/matt/Documents/Repositories/spark/dist/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py",
> line 363, in send_command
> File
> "/home/matt/Documents/Repositories/spark/dist/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py",
> line 472, in send_command
> File "/usr/lib64/python2.7/socket.py", line 430, in readline
> data = recv(1)
> KeyboardInterrupt
--
This message was sent by Atlassian JIRA
(v6.2#6252)
