Re: --packages configuration equivalent item name?

[Russell Jurney]

Thanks! These aren't in the docs, I will make a JIRA to add them.

On Monday, April 4, 2016, Saisai Shao  wrote:

> spark.jars.ivy, spark.jars.packages, spark.jars.excludes is the
> configurations you can use.
>
> Thanks
> Saisai
>
> On Sun, Apr 3, 2016 at 1:59 AM, Russell Jurney  > wrote:
>
>> Thanks, Andy!
>>
>> On Mon, Mar 28, 2016 at 8:44 AM, Andy Davidson <
>> a...@santacruzintegration.com
>> > wrote:
>>
>>> Hi Russell
>>>
>>> I use Jupyter python notebooks a lot. Here is how I start the server
>>>
>>> set -x # turn debugging on
>>>
>>> #set +x # turn debugging off
>>>
>>>
>>> # https://github.com/databricks/spark-csv
>>>
>>> # http://spark-packages.org/package/datastax/spark-cassandra-connector
>>>
>>> #
>>> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md
>>>
>>> #
>>> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md#pyspark-with-data-frames
>>>
>>>
>>> # packages are ',' seperate with no white space
>>>
>>> extraPkgs="--packages
>>> com.databricks:spark-csv_2.11:1.3.0,datastax:spark-cassandra-connector:1.6.0-M1-s_2.10"
>>>
>>>
>>> export PYSPARK_PYTHON=python3
>>>
>>> export PYSPARK_DRIVER_PYTHON=python3
>>>
>>> IPYTHON_OPTS=notebook $SPARK_ROOT/bin/pyspark $extraPkgs --conf
>>> spark.cassandra.connection.host=
>>> ec2-54-153-102-232.us-west-1.compute.amazonaws.com $*
>>>
>>>
>>>
>>> From: Russell Jurney >> >
>>> Date: Sunday, March 27, 2016 at 7:22 PM
>>> To: "user @spark" >> >
>>> Subject: --packages configuration equivalent item name?
>>>
>>> I run PySpark with CSV support like so: IPYTHON=1 pyspark --packages
>>> com.databricks:spark-csv_2.10:1.4.0
>>>
>>> I don't want to type this --packages argument each time. Is there a
>>> config item for --packages? I can't find one in the reference at
>>> http://spark.apache.org/docs/latest/configuration.html
>>>
>>> If there is no way to do this, please let me know so I can make a JIRA
>>> for this feature.
>>>
>>> Thanks!
>>> --
>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com
>>>  relato.io
>>>
>>>
>>
>>
>> --
>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com
>>  relato.io
>>
>
>

-- 
Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io

Re: --packages configuration equivalent item name?

[Saisai Shao]

spark.jars.ivy, spark.jars.packages, spark.jars.excludes is the
configurations you can use.

Thanks
Saisai

On Sun, Apr 3, 2016 at 1:59 AM, Russell Jurney 
wrote:

> Thanks, Andy!
>
> On Mon, Mar 28, 2016 at 8:44 AM, Andy Davidson <
> a...@santacruzintegration.com> wrote:
>
>> Hi Russell
>>
>> I use Jupyter python notebooks a lot. Here is how I start the server
>>
>> set -x # turn debugging on
>>
>> #set +x # turn debugging off
>>
>>
>> # https://github.com/databricks/spark-csv
>>
>> # http://spark-packages.org/package/datastax/spark-cassandra-connector
>>
>> #
>> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md
>>
>> #
>> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md#pyspark-with-data-frames
>>
>>
>> # packages are ',' seperate with no white space
>>
>> extraPkgs="--packages
>> com.databricks:spark-csv_2.11:1.3.0,datastax:spark-cassandra-connector:1.6.0-M1-s_2.10"
>>
>>
>> export PYSPARK_PYTHON=python3
>>
>> export PYSPARK_DRIVER_PYTHON=python3
>>
>> IPYTHON_OPTS=notebook $SPARK_ROOT/bin/pyspark $extraPkgs --conf
>> spark.cassandra.connection.host=
>> ec2-54-153-102-232.us-west-1.compute.amazonaws.com $*
>>
>>
>>
>> From: Russell Jurney 
>> Date: Sunday, March 27, 2016 at 7:22 PM
>> To: "user @spark" 
>> Subject: --packages configuration equivalent item name?
>>
>> I run PySpark with CSV support like so: IPYTHON=1 pyspark --packages
>> com.databricks:spark-csv_2.10:1.4.0
>>
>> I don't want to type this --packages argument each time. Is there a
>> config item for --packages? I can't find one in the reference at
>> http://spark.apache.org/docs/latest/configuration.html
>>
>> If there is no way to do this, please let me know so I can make a JIRA
>> for this feature.
>>
>> Thanks!
>> --
>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io
>>
>>
>
>
> --
> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io
>

Re: --packages configuration equivalent item name?

[Russell Jurney]

Thanks, Andy!

On Mon, Mar 28, 2016 at 8:44 AM, Andy Davidson <
a...@santacruzintegration.com> wrote:

> Hi Russell
>
> I use Jupyter python notebooks a lot. Here is how I start the server
>
> set -x # turn debugging on
>
> #set +x # turn debugging off
>
>
> # https://github.com/databricks/spark-csv
>
> # http://spark-packages.org/package/datastax/spark-cassandra-connector
>
> #
> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md
>
> #
> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md#pyspark-with-data-frames
>
>
> # packages are ',' seperate with no white space
>
> extraPkgs="--packages
> com.databricks:spark-csv_2.11:1.3.0,datastax:spark-cassandra-connector:1.6.0-M1-s_2.10"
>
>
> export PYSPARK_PYTHON=python3
>
> export PYSPARK_DRIVER_PYTHON=python3
>
> IPYTHON_OPTS=notebook $SPARK_ROOT/bin/pyspark $extraPkgs --conf
> spark.cassandra.connection.host=
> ec2-54-153-102-232.us-west-1.compute.amazonaws.com $*
>
>
>
> From: Russell Jurney 
> Date: Sunday, March 27, 2016 at 7:22 PM
> To: "user @spark" 
> Subject: --packages configuration equivalent item name?
>
> I run PySpark with CSV support like so: IPYTHON=1 pyspark --packages
> com.databricks:spark-csv_2.10:1.4.0
>
> I don't want to type this --packages argument each time. Is there a config
> item for --packages? I can't find one in the reference at
> http://spark.apache.org/docs/latest/configuration.html
>
> If there is no way to do this, please let me know so I can make a JIRA for
> this feature.
>
> Thanks!
> --
> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io
>
>


-- 
Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io

Re: --packages configuration equivalent item name?

[Andy Davidson]

Hi Russell

I use Jupyter python notebooks a lot. Here is how I start the server

set -x # turn debugging on

#set +x # turn debugging off



# https://github.com/databricks/spark-csv

# http://spark-packages.org/package/datastax/spark-cassandra-connector

#https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_py
thon.md

# 
https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_pyt
hon.md#pyspark-with-data-frames



# packages are ',' seperate with no white space

extraPkgs="--packages
com.databricks:spark-csv_2.11:1.3.0,datastax:spark-cassandra-connector:1.6.0
-M1-s_2.10"



export PYSPARK_PYTHON=python3

export PYSPARK_DRIVER_PYTHON=python3

IPYTHON_OPTS=notebook $SPARK_ROOT/bin/pyspark $extraPkgs --conf
spark.cassandra.connection.host=ec2-54-153-102-232.us-west-1.compute.amazona
ws.com $*



From:  Russell Jurney 
Date:  Sunday, March 27, 2016 at 7:22 PM
To:  "user @spark" 
Subject:  --packages configuration equivalent item name?

> I run PySpark with CSV support like so: IPYTHON=1 pyspark --packages
> com.databricks:spark-csv_2.10:1.4.0
> 
> I don't want to type this --packages argument each time. Is there a config
> item for --packages? I can't find one in the reference at
> http://spark.apache.org/docs/latest/configuration.html
> 
> If there is no way to do this, please let me know so I can make a JIRA for
> this feature.
> 
> Thanks!
> -- 
> Russell Jurney twitter.com/rjurney 
> russell.jur...@gmail.com relato.io