Hi guys,
I would like to be able to set the umask settings for when I am writing a
parquet file in spark through Zeppelin. I can set the spark settings:
*
spark.sparkContext.hadoopConfiguration.set("fs.permissions.umask-mode", "007")
Unfortunately, when it writes the parent folder and the _success files use the
OS file system which I can't change from Zeppelin. If I am running spark shell,
I can set it in my .profile folder and it will work. However, we run dockerised
Zeppelin and it will always revert to the global umask.
Is there a way to fix this?
Cheers,
Anton
Anton Bubna-Litic
QUANTIUM
Level 25, 8-12 Chifley Square
Sydney NSW 2000
Australia
T: +61 2 8222 3585
W: quantium.com.au<http://www.quantium.com>
________________________________
linkedin.com/company/quantium<http://www.linkedin.com/company/quantium>
facebook.com/QuantiumAustralia<http://www.facebook.com/QuantiumAustralia>
twitter.com/QuantiumAU<http://www.twitter.com/QuantiumAU>
The contents of this email, including attachments, may be confidential
information. If you are not the intended recipient, any use, disclosure or
copying of the information is unauthorised. If you have received this email in
error, we would be grateful if you would notify us immediately by email reply,
phone (+ 61 2 9292 6400) or fax (+ 61 2 9292 6444) and delete the message from
your system.
