I am having trouble setting up my python3 virtualenv. I created a virtualenv spark-2.3.0¹ Installed pyspark using pip how ever I am not able to import pyspark.sql.functions. I get ³unresolved import² when I try to import col() and lit()
from pyspark.sql.functions import * I found if I download spark from apache and set SPARK_ROOT I can get my juypter notebook to work. This is a very error prone work around. I am having simiilar problem with my eclipse pyDev virtualenv Any suggestions would be greatly appreciated Andy # pip show in virtualenv (spark-2.3.0) $ pip show pyspark Name: pyspark Version: 2.3.0 Summary: Apache Spark Python API Home-page: https://github.com/apache/spark/tree/master/python Author: Spark Developers Author-email: d...@spark.apache.org License: http://www.apache.org/licenses/LICENSE-2.0 Location: /Users/foo/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site-packages Requires: py4j (spark-2.3.0) $ (spark-2.3.0) $ ls ~/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site-packages/pyspark/sql/fu nctions.py ~/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site-packages/pyspark/sql/fu nctions.py # Jupyter Notebook Export SPARK_ROOT=~/workSpace/spark/spark-2.3.0-bin-hadoop2.7 Eclipse pyDev virtual ENV
--------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
