On Tue, Sep 30, 2014 at 10:14 PM, Rick Richardson
rick.richard...@gmail.com wrote:
I am experiencing significant logging spam when running PySpark in IPython
Notebok
Exhibit A: http://i.imgur.com/BDP0R2U.png
I have taken into consideration advice from:
Thanks for your reply. Unfortunately changing the log4j.properties within
SPARK_HOME/conf has no effect on pyspark for me. When I change it in the
master or workers the log changes have the desired effect, but pyspark
seems to ignore them. I have changed the levels to WARN, changed the
appender
How do you setup IPython to access pyspark in notebook?
I did as following, it worked for me:
$ export SPARK_HOME=/opt/spark-1.1.0/
$ export
PYTHONPATH=/opt/spark-1.1.0/python:/opt/spark-1.1.0/python/lib/py4j-0.8.2.1-src.zip
$ ipython notebook
All the logging will go into console (not in
I was starting PySpark as a profile within IPython Notebook as per:
http://blog.cloudera.com/blog/2014/08/how-to-use-ipython-notebook-with-apache-spark/
My setup looks like:
import os
import sys
spark_home = os.environ.get('SPARK_HOME', None)
if not spark_home:
raise ValueError('SPARK_HOME
Here is the other relevant bit of my set-up:
MASTER=spark://sparkmaster:7077
IPYTHON_OPTS=notebook --pylab inline --ip=0.0.0.0
CASSANDRA_NODES=cassandra1|cassandra2|cassandra3
PYSPARK_SUBMIT_ARGS=--master $MASTER --deploy-mode client --num-executors
6 --executor-memory 1g --executor-cores 1
I found the problem. I was manually constructing the CLASSPATH and
SPARK_CLASSPATH because I needed jars for running the cassandra lib.
For some reason that I cannot explain, it was this that was causing the
issue. Maybe one of the jars had a log4j.properties rolled up in it?
I removed almost
I am experiencing significant logging spam when running PySpark in IPython
Notebok
Exhibit A: http://i.imgur.com/BDP0R2U.png
I have taken into consideration advice from:
http://apache-spark-user-list.1001560.n3.nabble.com/Disable-all-spark-logging-td1960.html
also
