[
https://issues.apache.org/jira/browse/SPARK-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell updated SPARK-1565:
-----------------------------------
Description:
Example:
https://github.com/pwendell/spark/commit/3fdad8ada28c1cef9b3b367993057de22295b8ed
Now that we have the spark-submit script we should make some changes to the
examples:
1. We should always create a SparkConf that sets the app name and use the
SparkContext constructor that accepts a conf.
2. We shouldn't set the master based on a command line argument, instead users
can do this on their own with spark-submit.
3. The examples projects should mark spark-core/streaming as a "provided"
dependency, so that the examples assembly jar does not include spark.
Then users can launch examples like this:
{code}
./bin/spark-submit
examples/target/scala-2.10/spark-examples-assembly-1.0.0-SNAPSHOT.jar \
--class org.apache.spark.examples.SparkPi \
--arg 1000
{code}
was:
https://github.com/pwendell/spark/commit/3fdad8ada28c1cef9b3b367993057de22295b8ed
Now that we have the spark-submit script we should make some changes to the
examples:
1. We should always create a SparkConf that sets the app name and use the
SparkContext constructor that accepts a conf.
2. We shouldn't set the master based on a command line argument, instead users
can do this on their own with spark-submit.
3. The examples projects should mark spark-core/streaming as a "provided"
dependency, so that the examples assembly jar does not include spark.
Then users can launch examples like this:
{code}
./bin/spark-submit
examples/target/scala-2.10/spark-examples-assembly-1.0.0-SNAPSHOT.jar \
--class org.apache.spark.examples.SparkPi \
--arg 1000
{code}
> Spark examples should be changed given spark-submit
> ---------------------------------------------------
>
> Key: SPARK-1565
> URL: https://issues.apache.org/jira/browse/SPARK-1565
> Project: Spark
> Issue Type: Improvement
> Components: Examples
> Reporter: Patrick Wendell
> Assignee: Prashant Sharma
> Fix For: 1.0.0
>
>
> Example:
> https://github.com/pwendell/spark/commit/3fdad8ada28c1cef9b3b367993057de22295b8ed
> Now that we have the spark-submit script we should make some changes to the
> examples:
> 1. We should always create a SparkConf that sets the app name and use the
> SparkContext constructor that accepts a conf.
> 2. We shouldn't set the master based on a command line argument, instead
> users can do this on their own with spark-submit.
> 3. The examples projects should mark spark-core/streaming as a "provided"
> dependency, so that the examples assembly jar does not include spark.
> Then users can launch examples like this:
> {code}
> ./bin/spark-submit
> examples/target/scala-2.10/spark-examples-assembly-1.0.0-SNAPSHOT.jar \
> --class org.apache.spark.examples.SparkPi \
> --arg 1000
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
