Thanks! These aren't in the docs, I will make a JIRA to add them. On Monday, April 4, 2016, Saisai Shao <sai.sai.s...@gmail.com> wrote:
> spark.jars.ivy, spark.jars.packages, spark.jars.excludes is the > configurations you can use. > > Thanks > Saisai > > On Sun, Apr 3, 2016 at 1:59 AM, Russell Jurney <russell.jur...@gmail.com > <javascript:_e(%7B%7D,'cvml','russell.jur...@gmail.com');>> wrote: > >> Thanks, Andy! >> >> On Mon, Mar 28, 2016 at 8:44 AM, Andy Davidson < >> a...@santacruzintegration.com >> <javascript:_e(%7B%7D,'cvml','a...@santacruzintegration.com');>> wrote: >> >>> Hi Russell >>> >>> I use Jupyter python notebooks a lot. Here is how I start the server >>> >>> set -x # turn debugging on >>> >>> #set +x # turn debugging off >>> >>> >>> # https://github.com/databricks/spark-csv >>> >>> # http://spark-packages.org/package/datastax/spark-cassandra-connector >>> >>> # >>> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md >>> >>> # >>> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md#pyspark-with-data-frames >>> >>> >>> # packages are ',' seperate with no white space >>> >>> extraPkgs="--packages >>> com.databricks:spark-csv_2.11:1.3.0,datastax:spark-cassandra-connector:1.6.0-M1-s_2.10" >>> >>> >>> export PYSPARK_PYTHON=python3 >>> >>> export PYSPARK_DRIVER_PYTHON=python3 >>> >>> IPYTHON_OPTS=notebook $SPARK_ROOT/bin/pyspark $extraPkgs --conf >>> spark.cassandra.connection.host= >>> ec2-54-153-102-232.us-west-1.compute.amazonaws.com $* >>> >>> >>> >>> From: Russell Jurney <russell.jur...@gmail.com >>> <javascript:_e(%7B%7D,'cvml','russell.jur...@gmail.com');>> >>> Date: Sunday, March 27, 2016 at 7:22 PM >>> To: "user @spark" <user@spark.apache.org >>> <javascript:_e(%7B%7D,'cvml','user@spark.apache.org');>> >>> Subject: --packages configuration equivalent item name? >>> >>> I run PySpark with CSV support like so: IPYTHON=1 pyspark --packages >>> com.databricks:spark-csv_2.10:1.4.0 >>> >>> I don't want to type this --packages argument each time. Is there a >>> config item for --packages? I can't find one in the reference at >>> http://spark.apache.org/docs/latest/configuration.html >>> >>> If there is no way to do this, please let me know so I can make a JIRA >>> for this feature. >>> >>> Thanks! >>> -- >>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>> <javascript:_e(%7B%7D,'cvml','russell.jur...@gmail.com');> relato.io >>> >>> >> >> >> -- >> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >> <javascript:_e(%7B%7D,'cvml','russell.jur...@gmail.com');> relato.io >> > > -- Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io
