Exactly, but it's only fixed for the next spark version. Is there any work
around for version 1.5.2?
On Wed, Dec 16, 2015 at 4:36 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> This seems related:
> [SPARK-10123][DEPLOY] Support specifying deploy mode from configuration
>
> FYI
>
> On Wed, Dec 16, 2015 at 7:31 AM, Saiph Kappa <saiph.ka...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I have a client application running on host0 that is launching multiple
>> drivers on multiple remote standalone spark clusters (each cluster is
>> running on a single machine):
>>
>> «
>> ...
>>
>> List("host1", "host2" , "host3").foreach(host => {
>>
>> val sparkConf = new SparkConf()
>> sparkConf.setAppName("App")
>>
>> sparkConf.set("spark.driver.memory", "4g")
>> sparkConf.set("spark.executor.memory", "4g")
>> sparkConf.set("spark.driver.maxResultSize", "4g")
>> sparkConf.set("spark.serializer",
>> "org.apache.spark.serializer.KryoSerializer")
>> sparkConf.set("spark.executor.extraJavaOptions", " -XX:+UseCompressedOops
>> -XX:+UseConcMarkSweepGC " +
>> "-XX:+AggressiveOpts -XX:FreqInlineSize=300 -XX:MaxInlineSize=300 ")
>>
>> sparkConf.setMaster(s"spark://$host:7077")
>>
>> val rawStreams = (1 to source.parallelism).map(_ =>
>> ssc.textFileStream("/home/user/data/")).toArray
>> val rawStream = ssc.union(rawStreams)
>> rawStream.count.map(c => s"Received $c records.").print()
>>
>> }
>> ...
>>
>> »
>>
>> The problem is that I'm getting an error message saying that the directory
>> "/home/user/data/" does not exist.
>> In fact, this directory only exists in host1, host2 and host3 and not in
>> host0.
>> But since I'm launching the driver to host1..3 I thought data would be
>> fetched from those machines.
>>
>> I'm also trying to avoid using the spark submit script, and couldn't find
>> the configuration parameter to specify the deploy mode.
>>
>> Is there any way to specify the deploy mode through configuration parameter?
>>
>>
>> Thanks.
>>
>>
>
