Here is the other relevant bit of my set-up: MASTER=spark://sparkmaster:7077 IPYTHON_OPTS="notebook --pylab inline --ip=0.0.0.0" CASSANDRA_NODES="cassandra1|cassandra2|cassandra3" PYSPARK_SUBMIT_ARGS="--master $MASTER --deploy-mode client --num-executors 6 --executor-memory 1g --executor-cores 1" ipython notebook --profile=pyspark
On Wed, Oct 1, 2014 at 3:41 PM, Rick Richardson <rick.richard...@gmail.com> wrote: > I was starting PySpark as a profile within IPython Notebook as per: > > http://blog.cloudera.com/blog/2014/08/how-to-use-ipython-notebook-with-apache-spark/ > > > My setup looks like: > import os > import sys > > spark_home = os.environ.get('SPARK_HOME', None) > if not spark_home: > raise ValueError('SPARK_HOME environment variable is not set') > sys.path.insert(0, os.path.join(spark_home, 'python')) > sys.path.insert(0, os.path.join(spark_home, > 'python/lib/py4j-0.8.2.1-src.zip')) > execfile(os.path.join(spark_home, 'python/pyspark/shell.py')) > > I also have some code to expand all of the jars (and the log4j.property) > in SPARK_HOME and CASSANDRA_HOME and add them to the SPARK_CLASSPATH > > > I'll try your launch method and see how that goes. > > On Wed, Oct 1, 2014 at 3:31 PM, Davies Liu <dav...@databricks.com> wrote: > >> How do you setup IPython to access pyspark in notebook? >> >> I did as following, it worked for me: >> >> $ export SPARK_HOME=/opt/spark-1.1.0/ >> $ export >> PYTHONPATH=/opt/spark-1.1.0/python:/opt/spark-1.1.0/python/lib/py4j-0.8.2.1-src.zip >> $ ipython notebook >> >> All the logging will go into console (not in notebook), >> >> If you want to reduce the logging in console, you should change >> /opt/spark-1.1.0/conf/log4j.properties >> >> log4j.rootCategory=WARN, console >> og4j.logger.org.apache.spark=WARN >> >> >> On Wed, Oct 1, 2014 at 11:49 AM, Rick Richardson >> <rick.richard...@gmail.com> wrote: >> > Thanks for your reply. Unfortunately changing the log4j.properties >> within >> > SPARK_HOME/conf has no effect on pyspark for me. When I change it in >> the >> > master or workers the log changes have the desired effect, but pyspark >> seems >> > to ignore them. I have changed the levels to WARN, changed the >> appender to >> > rolling file, or removed it entirely, all with the same results. >> > >> > On Wed, Oct 1, 2014 at 1:49 PM, Davies Liu <dav...@databricks.com> >> wrote: >> >> >> >> On Tue, Sep 30, 2014 at 10:14 PM, Rick Richardson >> >> <rick.richard...@gmail.com> wrote: >> >> > I am experiencing significant logging spam when running PySpark in >> >> > IPython >> >> > Notebok >> >> > >> >> > Exhibit A: http://i.imgur.com/BDP0R2U.png >> >> > >> >> > I have taken into consideration advice from: >> >> > >> >> > >> http://apache-spark-user-list.1001560.n3.nabble.com/Disable-all-spark-logging-td1960.html >> >> > >> >> > also >> >> > >> >> > >> >> > >> http://stackoverflow.com/questions/25193488/how-to-turn-off-info-logging-in-pyspark >> >> > >> >> > I have only one log4j.properties it is in /opt/spark-1.1.0/conf >> >> > >> >> > Just before I launch IPython Notebook with a pyspark profile, I add >> the >> >> > dir >> >> > and the properties file directly to CLASSPATH and SPARK_CLASSPATH env >> >> > vars >> >> > (as you can also see from the png) >> >> > >> >> > I still haven't been able to make any change which disables this >> >> > infernal >> >> > debug output. >> >> > >> >> > Any ideas (WAGs, Solutions, commiserating) would be greatly >> >> > appreciated. >> >> > >> >> > --- >> >> > >> >> > My log4j.properties: >> >> > >> >> > log4j.rootCategory=INFO, console >> >> > log4j.appender.console=org.apache.log4j.ConsoleAppender >> >> > log4j.appender.console.layout=org.apache.log4j.PatternLayout >> >> > log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd >> HH:mm:ss} %p >> >> > %c{1}: %m%n >> >> >> >> You should change log4j.rootCategory to WARN, console >> >> >> >> > # Change this to set Spark log level >> >> > log4j.logger.org.apache.spark=INFO >> >> > >> >> > # Silence akka remoting >> >> > log4j.logger.Remoting=WARN >> >> > >> >> > # Ignore messages below warning level from Jetty, because it's a bit >> >> > verbose >> >> > log4j.logger.org.eclipse.jetty=WARN >> >> > >> >> > >> > >> > >> > >> > >> > -- >> > >> > >> > “Science is the great antidote to the poison of enthusiasm and >> > superstition.” -- Adam Smith >> > >> > > > > -- > > > “Science is the great antidote to the poison of enthusiasm and > superstition.” -- Adam Smith > > -- “Science is the great antidote to the poison of enthusiasm and superstition.” -- Adam Smith
