Thanks, Andy! On Mon, Mar 28, 2016 at 8:44 AM, Andy Davidson < a...@santacruzintegration.com> wrote:
> Hi Russell > > I use Jupyter python notebooks a lot. Here is how I start the server > > set -x # turn debugging on > > #set +x # turn debugging off > > > # https://github.com/databricks/spark-csv > > # http://spark-packages.org/package/datastax/spark-cassandra-connector > > # > https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md > > # > https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md#pyspark-with-data-frames > > > # packages are ',' seperate with no white space > > extraPkgs="--packages > com.databricks:spark-csv_2.11:1.3.0,datastax:spark-cassandra-connector:1.6.0-M1-s_2.10" > > > export PYSPARK_PYTHON=python3 > > export PYSPARK_DRIVER_PYTHON=python3 > > IPYTHON_OPTS=notebook $SPARK_ROOT/bin/pyspark $extraPkgs --conf > spark.cassandra.connection.host= > ec2-54-153-102-232.us-west-1.compute.amazonaws.com $* > > > > From: Russell Jurney <russell.jur...@gmail.com> > Date: Sunday, March 27, 2016 at 7:22 PM > To: "user @spark" <user@spark.apache.org> > Subject: --packages configuration equivalent item name? > > I run PySpark with CSV support like so: IPYTHON=1 pyspark --packages > com.databricks:spark-csv_2.10:1.4.0 > > I don't want to type this --packages argument each time. Is there a config > item for --packages? I can't find one in the reference at > http://spark.apache.org/docs/latest/configuration.html > > If there is no way to do this, please let me know so I can make a JIRA for > this feature. > > Thanks! > -- > Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io > > -- Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io
