[ 
https://issues.apache.org/jira/browse/SPARK-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249656#comment-15249656
 ] 

Paul Shearer edited comment on SPARK-13973 at 4/20/16 11:36 AM:
----------------------------------------------------------------

In my development experience and those I know, the notebook is not the only or 
even the main use case for IPython. It's just the most visible because it's 
easy to make slides and such. IPython, without the notebook, is a productive 
general-purpose programming environment for data scientists.

Just to get us on the same page - pyspark is a python shell with SparkContext 
pre-loaded. The python shell can be either default python shell or the enhanced 
IPython shell. The latter offers tab completion of object attributes, nicely 
formatted tracebacks, "magic" macros for executing pasted code snippets, 
scripting, debugging, and accessing help/docstrings. It is a separate entity 
from the notebook and long predates the notebook. 

Most working scientists and analysts I know, including myself, use the IPython 
shell much more than the notebook. The notebook is more of a presentation and 
exploratory analysis tool, while the IPython shell is better for general 
programming / power users.

And yes, not overriding `PYSPARK_DRIVER_PYTHON` would avoid breaking my setup. 
I found it easy enough just to comment out `IPYTHON=1` once I realized that was 
the problem, so alternatively you could print some alert that this is 
deprecated and tell people how to change it, so you don't have to keep 
supporting it.


was (Author: pshearer):
Just to get us on the same page - pyspark is a python shell with SparkContext 
pre-loaded. The python shell can be either default python shell or the enhanced 
IPython shell. The latter offers tab completion of object attributes, nicely 
formatted tracebacks, "magic" macros for executing pasted code snippets, 
scripting, debugging, and accessing help/docstrings. It is a separate entity 
from the notebook and long predates the notebook. 

Most working scientists and analysts I know, including myself, use the IPython 
shell much more than the notebook. The notebook is more of a presentation and 
exploratory analysis tool, while the IPython shell is better for general 
programming / power users.

And yes, not overriding `PYSPARK_DRIVER_PYTHON` would avoid breaking my setup. 
I found it easy enough just to comment out `IPYTHON=1` once I realized that was 
the problem, so alternatively you could print some alert that this is 
deprecated and tell people how to change it, so you don't have to keep 
supporting it.

> `ipython notebook` is going away...
> -----------------------------------
>
>                 Key: SPARK-13973
>                 URL: https://issues.apache.org/jira/browse/SPARK-13973
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>         Environment: spark-1.6.1-bin-hadoop2.6
> Anaconda2-2.5.0-Linux-x86_64
>            Reporter: Bogdan Pirvu
>            Assignee: Rekha Joshi
>            Priority: Trivial
>             Fix For: 2.0.0
>
>
> Starting {{pyspark}} with following environment variables:
> {code:none}
> export IPYTHON=1
> export IPYTHON_OPTS="notebook --no-browser"
> {code}
> yields this warning
> {code:none}
> [TerminalIPythonApp] WARNING | Subcommand `ipython notebook` is deprecated 
> and will be removed in future versions.
> [TerminalIPythonApp] WARNING | You likely want to use `jupyter notebook`... 
> continue in 5 sec. Press Ctrl-C to quit now.
> {code}
> Changing line 52 from
> {code:none}
> PYSPARK_DRIVER_PYTHON="ipython"
> {code}
> to
> {code:none}
> PYSPARK_DRIVER_PYTHON="jupyter"
> {code}
> in https://github.com/apache/spark/blob/master/bin/pyspark works for me to 
> solve this issue, but I'm not sure if it's sustainable as I'm not familiar 
> with the rest of the code...
> This is the relevant part of my Python environment:
> {code:none}
> ipython                   4.1.2                    py27_0  
> ipython-genutils          0.1.0                     <pip>
> ipython_genutils          0.1.0                    py27_0  
> ipywidgets                4.1.1                    py27_0  
> ...
> jupyter                   1.0.0                    py27_1  
> jupyter-client            4.2.1                     <pip>
> jupyter-console           4.1.1                     <pip>
> jupyter-core              4.1.0                     <pip>
> jupyter_client            4.2.1                    py27_0  
> jupyter_console           4.1.1                    py27_0  
> jupyter_core              4.1.0                    py27_0
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to