How do you setup IPython to access pyspark in notebook? I did as following, it worked for me:
$ export SPARK_HOME=/opt/spark-1.1.0/ $ export PYTHONPATH=/opt/spark-1.1.0/python:/opt/spark-1.1.0/python/lib/py4j-0.8.2.1-src.zip $ ipython notebook All the logging will go into console (not in notebook), If you want to reduce the logging in console, you should change /opt/spark-1.1.0/conf/log4j.properties log4j.rootCategory=WARN, console og4j.logger.org.apache.spark=WARN On Wed, Oct 1, 2014 at 11:49 AM, Rick Richardson <rick.richard...@gmail.com> wrote: > Thanks for your reply. Unfortunately changing the log4j.properties within > SPARK_HOME/conf has no effect on pyspark for me. When I change it in the > master or workers the log changes have the desired effect, but pyspark seems > to ignore them. I have changed the levels to WARN, changed the appender to > rolling file, or removed it entirely, all with the same results. > > On Wed, Oct 1, 2014 at 1:49 PM, Davies Liu <dav...@databricks.com> wrote: >> >> On Tue, Sep 30, 2014 at 10:14 PM, Rick Richardson >> <rick.richard...@gmail.com> wrote: >> > I am experiencing significant logging spam when running PySpark in >> > IPython >> > Notebok >> > >> > Exhibit A: http://i.imgur.com/BDP0R2U.png >> > >> > I have taken into consideration advice from: >> > >> > http://apache-spark-user-list.1001560.n3.nabble.com/Disable-all-spark-logging-td1960.html >> > >> > also >> > >> > >> > http://stackoverflow.com/questions/25193488/how-to-turn-off-info-logging-in-pyspark >> > >> > I have only one log4j.properties it is in /opt/spark-1.1.0/conf >> > >> > Just before I launch IPython Notebook with a pyspark profile, I add the >> > dir >> > and the properties file directly to CLASSPATH and SPARK_CLASSPATH env >> > vars >> > (as you can also see from the png) >> > >> > I still haven't been able to make any change which disables this >> > infernal >> > debug output. >> > >> > Any ideas (WAGs, Solutions, commiserating) would be greatly >> > appreciated. >> > >> > --- >> > >> > My log4j.properties: >> > >> > log4j.rootCategory=INFO, console >> > log4j.appender.console=org.apache.log4j.ConsoleAppender >> > log4j.appender.console.layout=org.apache.log4j.PatternLayout >> > log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p >> > %c{1}: %m%n >> >> You should change log4j.rootCategory to WARN, console >> >> > # Change this to set Spark log level >> > log4j.logger.org.apache.spark=INFO >> > >> > # Silence akka remoting >> > log4j.logger.Remoting=WARN >> > >> > # Ignore messages below warning level from Jetty, because it's a bit >> > verbose >> > log4j.logger.org.eclipse.jetty=WARN >> > >> > > > > > > -- > > > “Science is the great antidote to the poison of enthusiasm and > superstition.” -- Adam Smith > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
