Hi Andrew, Thanks a lot for your response. I am aware of the '--master' flag in the spark-submit command. However I would like to create the SparkContext inside my coding.
Maybe I should elaborate a little bit further: I would like to reuse e.g. the result of any Spark computation inside my code. Here is the SparkPi example: String[] jars = new String[1]; > > jars[0] = System.getProperty("user.dir") + >> "/target/SparkPi-1.0-SNAPSHOT.jar"; > > >> SparkConf conf = new SparkConf() > > .setAppName("JavaSparkPi") > > .setMaster("spark://SPARK_HOST:7077") > > .setJars(jars); > > JavaSparkContext sc = new JavaSparkContext(conf); > > >> int slices = (args.length == 1) ? Integer.parseInt(args[0]) : 2; > > int n = 1000000 * slices; > > List<Integer> l = new ArrayList<Integer>(n); > > for (int i = 0; i < n; i++) { > > l.add(i); > > } > > >> JavaRDD<Integer> dataSet = sc.parallelize(l, slices); > > >> int *count* = dataSet.map(new Function<Integer, Integer>() { > > @Override > > public Integer call(Integer integer) { > > double x = Math.random() * 2 - 1; > > double y = Math.random() * 2 - 1; > > return (x * x + y * y < 1) ? 1 : 0; > > } > > }).reduce(new Function2<Integer, Integer, Integer>() { > > @Override > > public Integer call(Integer integer, Integer integer2) { > > return integer + integer2; > > } > > }); > > System.out.println("Pi is roughly " + 4.0 * *count* / n); > > >> sc.stop(); > > As you can see, I can reuse the result (count) in my coding directly. So my goal would be to resuse this kind of implementation in YARN mode (client/cluster mode). However, I didn't really find a solution how to do that, since I always have to submit my Spark code via spark-submit. What if I want to run this code as part of a web application which renders the result as web page? -- Andreas On Tue, Aug 18, 2015 at 10:50 PM, Andrew Or <and...@databricks.com> wrote: > Hi Andreas, > > I believe the distinction is not between standalone and YARN mode, but > between client and cluster mode. > > In client mode, your Spark submit JVM runs your driver code. In cluster > mode, one of the workers (or NodeManagers if you're using YARN) in the > cluster runs your driver code. In the latter case, it doesn't really make > sense to call `setMaster` in your driver because Spark needs to know which > cluster you're submitting the application to. > > Instead, the recommended way is to set the master through the `--master` > flag in the command line, e.g. > > $ bin/spark-submit > --master spark://1.2.3.4:7077 > --class some.user.Clazz > --name "My app name" > --jars lib1.jar,lib2.jar > --deploy-mode cluster > app.jar > > Both YARN and standalone modes support client and cluster modes, and the > spark-submit script is the common interface through which you can launch > your application. In other words, you shouldn't have to do anything more > than providing a different value to `--master` to use YARN. > > -Andrew > > 2015-08-17 0:34 GMT-07:00 Andreas Fritzler <andreas.fritz...@gmail.com>: > >> Hi all, >> >> when runnig the Spark cluster in standalone mode I am able to create the >> Spark context from Java via the following code snippet: >> >> SparkConf conf = new SparkConf() >>> .setAppName("MySparkApp") >>> .setMaster("spark://SPARK_MASTER:7077") >>> .setJars(jars); >>> JavaSparkContext sc = new JavaSparkContext(conf); >> >> >> As soon as I'm done with my processing, I can just close it via >> >>> sc.stop(); >>> >> Now my question: Is the same also possible when running Spark on YARN? I >> currently don't see how this should be possible without submitting your >> application as a packaged jar file. Is there a way to get this kind of >> interactivity from within your Scala/Java code? >> >> Regards, >> Andrea >> > >