Spark-submit --conf spark.hadoop.fs.permissions.umask-mode=007 You may also set sticky bit on staging dir
Sent from my iPhone > On 26 Feb 2021, at 03:29, Bulldog20630405 <bulldog20630...@gmail.com> wrote: > > > > we have a spark cluster running on with multiple users... > when running with the user owning the cluster jobs run fine... however, when > trying to run pyspark with a different user it fails because the > .sparkStaging/application_* is written with 700 so the user cannot write to > that directory.... > > how to configure spark/yarn cluster so .sparkStaging is written as 770 > instead of 700 so the users with shared group can execute? > >
