This worked without setting other options: spark/bin/spark-submit --conf "spark.driver.extraJavaOptions=-Duser.language=en" test.py
Thank you again! Pietro > Il giorno 24 ott 2016, alle ore 17:18, Sean Owen <so...@cloudera.com> ha > scritto: > > I believe it will be too late to set it there, and these are JVM flags, not > app or Spark flags. See spark.driver.extraJavaOptions and likewise for the > executor. > > On Mon, Oct 24, 2016 at 4:04 PM Pietro Pugni <pietro.pu...@gmail.com > <mailto:pietro.pu...@gmail.com>> wrote: > Thank you! > > I tried again setting locale options in different ways but doesn’t propagate > to the JVM. I tested these strategies (alone and all together): > - bin/spark-submit --conf "spark.executor.extraJavaOptions=-Duser.language=en > -Duser.region=US -Duser.country=US -Duser.timezone=GMT” test.py > - spark = SparkSession \ > .builder \ > .appName("My app") \ > .config("spark.executor.extraJavaOptions", "-Duser.language=en > -Duser.region=US -Duser.country=US -Duser.timezone=GMT") \ > .config("user.country", "US") \ > .config("user.region", "US") \ > .config("user.language", "en") \ > .config("user.timezone", "GMT") \ > .config("-Duser.country", "US") \ > .config("-Duser.region", "US") \ > .config("-Duser.language", "en") \ > .config("-Duser.timezone", "GMT") \ > .getOrCreate() > - export JAVA_OPTS="-Duser.language=en -Duser.region=US -Duser.country=US > -Duser.timezone=GMT” > - export LANG="en_US.UTF-8” > > After running export LANG="en_US.UTF-8” from the same terminal session I use > to launch spark-submit, if I run locale command I get correct values: > LANG="en_US.UTF-8" > LC_COLLATE="en_US.UTF-8" > LC_CTYPE="en_US.UTF-8" > LC_MESSAGES="en_US.UTF-8" > LC_MONETARY="en_US.UTF-8" > LC_NUMERIC="en_US.UTF-8" > LC_TIME="en_US.UTF-8" > LC_ALL= > > While running my pyspark script, from the Spark UI, under Environment -> > Spark Properties the locale appear to be correctly set: > - user.country: US > - user.language: en > - user.region: US > - user.timezone: GMT > > but Environment -> System Properties still reports the System locale and not > the session locale I previously set: > - user.country: IT > - user.language: it > - user.timezone: Europe/Rome > > Am I wrong or the options don’t propagate to the JVM correctly? > >
