Hello everybody, I'm running a two node spark cluster on ec2, created using the provided scripts. I then ssh into the master and invoke "PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS='notebook --profile=pyspark' spark/bin/pyspark". This launches a spark notebook which has been instructed to listen to all interfaces, not only localhost. I then open my browser and start playing around.
All commands run fine as far as I've seen but there's an annoying problem: I cannot display matplotlib graphs in a cell, I get the following error "TclError: no display name and no $DISPLAY environment variable". I've searched the web and I've tried the following two approaches: 1. use -X to enable X11 forwarding: when I use this option I get no error, a slow execution time and no image at all 2. use matplotlib.use('agg'), no image but if I execute fig.savefig I can totally see the image being created. Has anybody have a similar problem? If so can you help me troubleshoot? Thanks, MD