Hello Sandy, Your suggestion does not work when I try it locally:
When I pass --conf "key=someValue" and then try to retrieve it like: SparkConf sparkConf = new SparkConf(); logger.info("* * * key ~~~> {}", sparkConf.get("key")); I get Exception in thread "main" java.util.NoSuchElementException: key And I think that's expected because the key is an arbitrary one, not necessarily a Spark configuration element. This is why I was passing it via --conf and retrieving System.getProperty("key") (which worked locally and in yarn-client mode but not in yarn-cluster mode). I'm surprised why I can't use it on the cluster while I can use it while local development and testing. Kind regards, Emre Sevinç http://www.bigindustries.be/ On Mon, Mar 23, 2015 at 6:15 PM, Sandy Ryza <sandy.r...@cloudera.com> wrote: > Hi Emre, > > The --conf property is meant to work with yarn-cluster mode. > System.getProperty("key") isn't guaranteed, but new SparkConf().get("key") > should. Does it not? > > -Sandy > > On Mon, Mar 23, 2015 at 8:39 AM, Emre Sevinc <emre.sev...@gmail.com> > wrote: > >> Hello, >> >> According to Spark Documentation at >> https://spark.apache.org/docs/1.2.1/submitting-applications.html : >> >> --conf: Arbitrary Spark configuration property in key=value format. >> For values that contain spaces wrap “key=value” in quotes (as shown). >> >> And indeed, when I use that parameter, in my Spark program I can retrieve >> the value of the key by using: >> >> System.getProperty("key"); >> >> This works when I test my program locally, and also in yarn-client mode, >> I can log the value of the key and see that it matches what I wrote in the >> command line, but it returns *null* when I submit the very same program in >> *yarn-cluster* mode. >> >> Why can't I retrieve the value of key given as --conf "key=value" when I >> submit my Spark application in *yarn-cluster* mode? >> >> Any ideas and/or workarounds? >> >> >> -- >> Emre Sevinç >> http://www.bigindustries.be/ >> >> > -- Emre Sevinc