spark.jars.ivy, spark.jars.packages, spark.jars.excludes is the configurations you can use.
Thanks Saisai On Sun, Apr 3, 2016 at 1:59 AM, Russell Jurney <russell.jur...@gmail.com> wrote: > Thanks, Andy! > > On Mon, Mar 28, 2016 at 8:44 AM, Andy Davidson < > a...@santacruzintegration.com> wrote: > >> Hi Russell >> >> I use Jupyter python notebooks a lot. Here is how I start the server >> >> set -x # turn debugging on >> >> #set +x # turn debugging off >> >> >> # https://github.com/databricks/spark-csv >> >> # http://spark-packages.org/package/datastax/spark-cassandra-connector >> >> # >> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md >> >> # >> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md#pyspark-with-data-frames >> >> >> # packages are ',' seperate with no white space >> >> extraPkgs="--packages >> com.databricks:spark-csv_2.11:1.3.0,datastax:spark-cassandra-connector:1.6.0-M1-s_2.10" >> >> >> export PYSPARK_PYTHON=python3 >> >> export PYSPARK_DRIVER_PYTHON=python3 >> >> IPYTHON_OPTS=notebook $SPARK_ROOT/bin/pyspark $extraPkgs --conf >> spark.cassandra.connection.host= >> ec2-54-153-102-232.us-west-1.compute.amazonaws.com $* >> >> >> >> From: Russell Jurney <russell.jur...@gmail.com> >> Date: Sunday, March 27, 2016 at 7:22 PM >> To: "user @spark" <user@spark.apache.org> >> Subject: --packages configuration equivalent item name? >> >> I run PySpark with CSV support like so: IPYTHON=1 pyspark --packages >> com.databricks:spark-csv_2.10:1.4.0 >> >> I don't want to type this --packages argument each time. Is there a >> config item for --packages? I can't find one in the reference at >> http://spark.apache.org/docs/latest/configuration.html >> >> If there is no way to do this, please let me know so I can make a JIRA >> for this feature. >> >> Thanks! >> -- >> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io >> >> > > > -- > Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io >
