[ https://issues.apache.org/jira/browse/SPARK-30860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun reassigned SPARK-30860: ------------------------------------- Assignee: Adam Binford > Different behavior between rolling and non-rolling event log > ------------------------------------------------------------ > > Key: SPARK-30860 > URL: https://issues.apache.org/jira/browse/SPARK-30860 > Project: Spark > Issue Type: Sub-task > Components: Spark Core > Affects Versions: 3.0.0 > Reporter: Adam Binford > Assignee: Adam Binford > Priority: Major > > When creating a rolling event log, the application directory is created with > a call to FileSystem.mkdirs, with the file permission 770. The default > behavior of HDFS is to set the permission of a file created with > FileSystem.create or FileSystem.mkdirs to (P & ^umask), where P is the > permission in the API call and umask is a system value set by > fs.permissions.umask-mode and defaults to 0022. This means, with default > settings, any mkdirs call can have at most 755 permissions, which causes > rolling event log directories to be created with 750 permissions. This causes > the history server to be unable to prune old applications if they are not run > by the same user running the history server. > This is not a problem for non-rolling logs, because it uses > SparkHadoopUtils.createFile for Hadoop 2 backward compatibility, and then > calls FileSystem.setPermission with 770 after the file has been created. > setPermission doesn't have the umask applied to it, so this works fine. > Obviously this could be fixed by changing fs.permissions.umask-mode, but I'm > not sure the reason that's set in the first place or if this would hurt > anything else. The main issue is there is different behavior between rolling > and non-rolling event logs that might want to be updated in this repo to be > consistent across each. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org