Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/14303
I don't know if this is the intended way to use PySpark (normally you would
use it through `spark-submit` or `pyspark`) - however if you want to use
PySpark this way you can see
https://github.com/sparklingpandas/sparklingpandas/blob/master/sparklingpandas/utils.py
to see one way how to setup the imports and then just do an `import pyspark` -
although it should be more or less equivalent with what you've posted for repro.
That being said I tried your repro steps with Python 2.7.11 & 3.4.3 on
2ae7b88a07140e012b6c60db3c4a2a8ca360c684 and it doesn't repro:
>>> import sys
>>> import os
>>> sys.path = [os.environ['SPARK_HOME'] +
"/python/lib/py4j-0.10.1-src.zip"] + sys.path
>>> sys.path = [os.environ['SPARK_HOME'] + "/python/"] + sys.path
>>> sys.path
['/home/holden/repos/spark/python/',
'/home/holden/repos/spark/python/lib/py4j-0.10.1-src.zip', '',
'/home/holden/miniconda2/lib/python27.zip',
'/home/holden/miniconda2/lib/python2.7',
'/home/holden/miniconda2/lib/python2.7/plat-linux2',
'/home/holden/miniconda2/lib/python2.7/lib-tk',
'/home/holden/miniconda2/lib/python2.7/lib-old',
'/home/holden/miniconda2/lib/python2.7/lib-dynload',
'/home/holden/.local/lib/python2.7/site-packages',
'/home/holden/miniconda2/lib/python2.7/site-packages',
'/home/holden/miniconda2/lib/python2.7/site-packages/setuptools-20.3-py2.7.egg']
>>> import pyspark
#delayedflightjuststartedtoboard
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
