Adam Binford created SPARK-30860:
------------------------------------
Summary: Different behavior between rolling and non-rolling event
log
Key: SPARK-30860
URL: https://issues.apache.org/jira/browse/SPARK-30860
Project: Spark
Issue Type: Bug
Components: Deploy
Affects Versions: 3.0.0
Reporter: Adam Binford
When creating a rolling event log, the application directory is created with a
call to FileSystem.mkdirs, with the file permission 770. The default behavior
of HDFS is to set the permission of a file created with FileSystem.create or
FileSystem.mkdirs to (P & ^umask), where P is the permission in the API call
and umask is a system value set by fs.permissions.umask-mode and defaults to
0022. This means, with default settings, any mkdirs call can have at most 755
permissions, which causes rolling event log directories to be created with 750
permissions. This causes the history server to be unable to prune old
applications if they are not run by the same user running the history server.
This is not a problem for non-rolling logs, because it uses
SparkHadoopUtils.createFile for Hadoop 2 backward compatibility, and then calls
FileSystem.setPermission with 770 after the file has been created.
setPermission doesn't have the umask applied to it, so this works fine.
Obviously this could be fixed by changing fs.permissions.umask-mode, but I'm
not sure the reason that's set in the first place or if this would hurt
anything else. The main issue is there is different behavior between rolling
and non-rolling event logs that might want to be updated in this repo to be
consistent across each.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
