I used to use spark 1.3.x before, and explore my data in ipython [3.2] notebook, which was very stable. but I came across an error
" Java gateway process exited before sending the driver its port number " my code is as bellow: ``` import pyspark from pyspark import SparkConf sc_conf = SparkConf() ### error occurs here ``` then I ask google for help, here is a answer on stackoverflow: http://stackoverflow.com/questions/30763951/spark-context-sc-not-defined/30851037#30851037 , it says: ``` One solution is adding pyspark-shell to the shell environment variable PYSPARK_SUBMIT_ARGS: export PYSPARK_SUBMIT_ARGS="--master local[2] pyspark-shell" There is a change in python/pyspark/java_gateway.py <https://github.com/apache/spark/commit/e3f315ac358dfe4f5b9705c3eac76e8b1e24f82a#diff-d88dad45f458b7ad9f1c6a63b07e3a28>, which requires PYSPARK_SUBMIT_ARGS includes pyspark-shell if a PYSPARK_SUBMIT_ARGS variable is set by a user. ``` then I change my PYSPARK_SUBMIT_ARGS from `--master spark:// 10.21.208.21:7077 --deploy-mode client` to `--master spark:// 10.21.208.21:7077 --deploy-mode client pyspark-shell` , it does works, but it raise another question, each time when I create sc in different notebooks, the spark app name is `pyspark-shell` even though I explicitily set the app name using SparkConf, that's really confused me these days. then My questions come: - How to start ipython notebook with spark integrated in spark 1.6.0; - why it works when I set `pyspark-shell` in PYSPARK_SUBMIT_ARGS when start ipython notebook with spark 1.6.0; - why it does not work when I explicitly set the app name using SparkConf; great thanks. -- *--------------------------------------* a spark lover, a quant, a developer and a good man. http://github.com/litaotao