Davies Liu created SPARK-9942:
---------------------------------
Summary: Broken pandas could crash PySpark SQL
Key: SPARK-9942
URL: https://issues.apache.org/jira/browse/SPARK-9942
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 1.4.1, 1.3.1, 1.5.0
Reporter: Davies Liu
Assignee: Davies Liu
Priority: Blocker
{code}
Error from python worker:
Traceback (most recent call last):
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 151, in _run_module_as_main
mod_name, loader, code, fname = _get_module_details(mod_name)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 101, in _get_module_details
loader = get_loader(mod_name)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pkgutil.py",
line 464, in get_loader
return find_loader(fullname)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pkgutil.py",
line 474, in find_loader
for importer in iter_importers(fullname):
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pkgutil.py",
line 430, in iter_importers
__import__(pkg)
File "/Users/davies/work/spark/python/pyspark/__init__.py", line 52, in
<module>
from pyspark.sql import SQLContext, HiveContext, SchemaRDD, Row
File "/Users/davies/work/spark/python/pyspark/sql/__init__.py", line 63, in
<module>
from pyspark.sql.context import SQLContext, HiveContext
File "/Users/davies/work/spark/python/pyspark/sql/context.py", line 40, in
<module>
import pandas
File "/Library/Python/2.7/site-packages/pandas/__init__.py", line 7, in
<module>
from pandas import hashtable, tslib, lib
File "pandas/src/numpy.pxd", line 157, in init pandas.hashtable
(pandas/hashtable.c:23654)
ValueError: numpy.dtype has the wrong size, try recompiling
PYTHONPATH was:
/Users/davies/work/spark/python/lib/pyspark.zip:/Users/davies/work/spark/python/lib/py4j-0.8.2.1-src.zip:/Users/davies/work/spark/sbin/../python/lib/py4j-0.8.2.1-src.zip:/Users/davies/work/spark/sbin/../python:/Users/davies/work/spark/sbin/../python/lib/py4j-0.8.2.1-src.zip:/Users/davies/work/spark/sbin/../python:
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
