Hello Sandy,
Your suggestion does not work when I try it locally:
When I pass
--conf "key=someValue"
and then try to retrieve it like:
SparkConf sparkConf = new SparkConf();
logger.info("* * * key ~~~> {}", sparkConf.get("key"));
I get
Exception in thread "main" java.util.NoSuchElementException: key
And I think that's expected because the key is an arbitrary one, not
necessarily a Spark configuration element. This is why I was passing it via
--conf and retrieving System.getProperty("key") (which worked locally and
in yarn-client mode but not in yarn-cluster mode). I'm surprised why I
can't use it on the cluster while I can use it while local development and
testing.
Kind regards,
Emre Sevinç
http://www.bigindustries.be/
On Mon, Mar 23, 2015 at 6:15 PM, Sandy Ryza <sandy.r...@cloudera.com> wrote:
> Hi Emre,
>
> The --conf property is meant to work with yarn-cluster mode.
> System.getProperty("key") isn't guaranteed, but new SparkConf().get("key")
> should. Does it not?
>
> -Sandy
>
> On Mon, Mar 23, 2015 at 8:39 AM, Emre Sevinc <emre.sev...@gmail.com>
> wrote:
>
>> Hello,
>>
>> According to Spark Documentation at
>> https://spark.apache.org/docs/1.2.1/submitting-applications.html :
>>
>> --conf: Arbitrary Spark configuration property in key=value format.
>> For values that contain spaces wrap “key=value” in quotes (as shown).
>>
>> And indeed, when I use that parameter, in my Spark program I can retrieve
>> the value of the key by using:
>>
>> System.getProperty("key");
>>
>> This works when I test my program locally, and also in yarn-client mode,
>> I can log the value of the key and see that it matches what I wrote in the
>> command line, but it returns *null* when I submit the very same program in
>> *yarn-cluster* mode.
>>
>> Why can't I retrieve the value of key given as --conf "key=value" when I
>> submit my Spark application in *yarn-cluster* mode?
>>
>> Any ideas and/or workarounds?
>>
>>
>> --
>> Emre Sevinç
>> http://www.bigindustries.be/
>>
>>
>
--
Emre Sevinc
