[jira] [Updated] (SPARK-30860) Different behavior between rolling and non-rolling event log
[ https://issues.apache.org/jira/browse/SPARK-30860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30860: -- Parent: SPARK-28594 Issue Type: Sub-task (was: Bug) > Different behavior between rolling and non-rolling event log > > > Key: SPARK-30860 > URL: https://issues.apache.org/jira/browse/SPARK-30860 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Adam Binford >Priority: Major > > When creating a rolling event log, the application directory is created with > a call to FileSystem.mkdirs, with the file permission 770. The default > behavior of HDFS is to set the permission of a file created with > FileSystem.create or FileSystem.mkdirs to (P & ^umask), where P is the > permission in the API call and umask is a system value set by > fs.permissions.umask-mode and defaults to 0022. This means, with default > settings, any mkdirs call can have at most 755 permissions, which causes > rolling event log directories to be created with 750 permissions. This causes > the history server to be unable to prune old applications if they are not run > by the same user running the history server. > This is not a problem for non-rolling logs, because it uses > SparkHadoopUtils.createFile for Hadoop 2 backward compatibility, and then > calls FileSystem.setPermission with 770 after the file has been created. > setPermission doesn't have the umask applied to it, so this works fine. > Obviously this could be fixed by changing fs.permissions.umask-mode, but I'm > not sure the reason that's set in the first place or if this would hurt > anything else. The main issue is there is different behavior between rolling > and non-rolling event logs that might want to be updated in this repo to be > consistent across each. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30860) Different behavior between rolling and non-rolling event log
[ https://issues.apache.org/jira/browse/SPARK-30860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-30860: - Priority: Major (was: Minor) > Different behavior between rolling and non-rolling event log > > > Key: SPARK-30860 > URL: https://issues.apache.org/jira/browse/SPARK-30860 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Adam Binford >Priority: Major > > When creating a rolling event log, the application directory is created with > a call to FileSystem.mkdirs, with the file permission 770. The default > behavior of HDFS is to set the permission of a file created with > FileSystem.create or FileSystem.mkdirs to (P & ^umask), where P is the > permission in the API call and umask is a system value set by > fs.permissions.umask-mode and defaults to 0022. This means, with default > settings, any mkdirs call can have at most 755 permissions, which causes > rolling event log directories to be created with 750 permissions. This causes > the history server to be unable to prune old applications if they are not run > by the same user running the history server. > This is not a problem for non-rolling logs, because it uses > SparkHadoopUtils.createFile for Hadoop 2 backward compatibility, and then > calls FileSystem.setPermission with 770 after the file has been created. > setPermission doesn't have the umask applied to it, so this works fine. > Obviously this could be fixed by changing fs.permissions.umask-mode, but I'm > not sure the reason that's set in the first place or if this would hurt > anything else. The main issue is there is different behavior between rolling > and non-rolling event logs that might want to be updated in this repo to be > consistent across each. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30860) Different behavior between rolling and non-rolling event log
[ https://issues.apache.org/jira/browse/SPARK-30860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-30860: - Component/s: (was: Deploy) Spark Core > Different behavior between rolling and non-rolling event log > > > Key: SPARK-30860 > URL: https://issues.apache.org/jira/browse/SPARK-30860 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Adam Binford >Priority: Minor > > When creating a rolling event log, the application directory is created with > a call to FileSystem.mkdirs, with the file permission 770. The default > behavior of HDFS is to set the permission of a file created with > FileSystem.create or FileSystem.mkdirs to (P & ^umask), where P is the > permission in the API call and umask is a system value set by > fs.permissions.umask-mode and defaults to 0022. This means, with default > settings, any mkdirs call can have at most 755 permissions, which causes > rolling event log directories to be created with 750 permissions. This causes > the history server to be unable to prune old applications if they are not run > by the same user running the history server. > This is not a problem for non-rolling logs, because it uses > SparkHadoopUtils.createFile for Hadoop 2 backward compatibility, and then > calls FileSystem.setPermission with 770 after the file has been created. > setPermission doesn't have the umask applied to it, so this works fine. > Obviously this could be fixed by changing fs.permissions.umask-mode, but I'm > not sure the reason that's set in the first place or if this would hurt > anything else. The main issue is there is different behavior between rolling > and non-rolling event logs that might want to be updated in this repo to be > consistent across each. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
