[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable
[ https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946874#comment-15946874 ] Arina Ielchiieva commented on DRILL-5391: - Zelaine, regarding the decision, with CTTAS introduction approach for creating files in Drill has changed, since we had to distinguish which files are temporary and which are not. Writers were not aware which files they created (fs.create(fileName)). To make solution configurable was decided to pass permission which should be applied to the folder rather that indicating if this is temp files or not (so if permission needs to be changed for some reason, it would be easily maintained). Having permission 755 / 644 for persistent files is common permission among systems and applications. Having this permission for Drill did not seem to break the anything at that time (well, I guess this Jira use case proves we were wrong). I think it a good idea for Drill to create files with permission (like it's done in Hive), though I guess to make this option configurable is what we have missed. For example, for views Drill also creates files with specified permission and it can be configurable. > CTAS: folder and file permission should be configurable > --- > > Key: DRILL-5391 > URL: https://issues.apache.org/jira/browse/DRILL-5391 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0, 1.10.0 > Environment: CentOS 7, HDP 2.4 >Reporter: Chua Tianxiang >Priority: Minor > Attachments: Drill-1-10.PNG, Drill-1-9.PNG > > > In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on > Drill 1.10, the same commands creates a folder with permission 775. Both > drills are started with root user, installed on the same servers and accesses > the same HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable
[ https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946831#comment-15946831 ] Arina Ielchiieva commented on DRILL-5391: - Not sure, why Drill 1.9 creates folders with 777 on your env. For example, for me on Linux or maprfs Drill 1.9 creates folders with 755. > CTAS: folder and file permission should be configurable > --- > > Key: DRILL-5391 > URL: https://issues.apache.org/jira/browse/DRILL-5391 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0, 1.10.0 > Environment: CentOS 7, HDP 2.4 >Reporter: Chua Tianxiang >Priority: Minor > Attachments: Drill-1-10.PNG, Drill-1-9.PNG > > > In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on > Drill 1.10, the same commands creates a folder with permission 775. Both > drills are started with root user, installed on the same servers and accesses > the same HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable
[ https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946758#comment-15946758 ] Chua Tianxiang commented on DRILL-5391: --- Arina, I got the following 755, umask 022. drwxr-xr-x - root hdfs 0 2017-03-29 08:58 /tmp/folder > CTAS: folder and file permission should be configurable > --- > > Key: DRILL-5391 > URL: https://issues.apache.org/jira/browse/DRILL-5391 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0, 1.10.0 > Environment: CentOS 7, HDP 2.4 >Reporter: Chua Tianxiang >Priority: Minor > Attachments: Drill-1-10.PNG, Drill-1-9.PNG > > > In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on > Drill 1.10, the same commands creates a folder with permission 775. Both > drills are started with root user, installed on the same servers and accesses > the same HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable
[ https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946750#comment-15946750 ] Arina Ielchiieva commented on DRILL-5391: - Chua, when you create folder on hdfs manually (hadoop fs -mkdir /some-dir) under the user who started the drillbit (root), with what permission directory is created? > CTAS: folder and file permission should be configurable > --- > > Key: DRILL-5391 > URL: https://issues.apache.org/jira/browse/DRILL-5391 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0, 1.10.0 > Environment: CentOS 7, HDP 2.4 >Reporter: Chua Tianxiang >Priority: Minor > Attachments: Drill-1-10.PNG, Drill-1-9.PNG > > > In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on > Drill 1.10, the same commands creates a folder with permission 775. Both > drills are started with root user, installed on the same servers and accesses > the same HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable
[ https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946396#comment-15946396 ] Chua Tianxiang commented on DRILL-5391: --- I've checked our configuration and can confirm that we have kept HDFS fs.permissions.umask-mode as default to 022. I've also checked that our dfs.namenode.acls.enabled setting is set to false. I'm not sure where else would this be configured. > CTAS: folder and file permission should be configurable > --- > > Key: DRILL-5391 > URL: https://issues.apache.org/jira/browse/DRILL-5391 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0, 1.10.0 > Environment: CentOS 7, HDP 2.4 >Reporter: Chua Tianxiang >Priority: Minor > Attachments: Drill-1-10.PNG, Drill-1-9.PNG > > > In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on > Drill 1.10, the same commands creates a folder with permission 775. Both > drills are started with root user, installed on the same servers and accesses > the same HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable
[ https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945522#comment-15945522 ] Zelaine Fong commented on DRILL-5391: - OK, last question :). Was there a reason you chose to use 775/664 vs using the default set by the underlying storage system? > CTAS: folder and file permission should be configurable > --- > > Key: DRILL-5391 > URL: https://issues.apache.org/jira/browse/DRILL-5391 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0, 1.10.0 > Environment: CentOS 7, HDP 2.4 >Reporter: Chua Tianxiang >Priority: Minor > Attachments: Drill-1-10.PNG, Drill-1-9.PNG > > > In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on > Drill 1.10, the same commands creates a folder with permission 775. Both > drills are started with root user, installed on the same servers and accesses > the same HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable
[ https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945485#comment-15945485 ] Arina Ielchiieva commented on DRILL-5391: - No, the default for HDFS is fs.permissions.umask-mode = 022 (775/644). I guess, in this particular case user has modified the defaults. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html > CTAS: folder and file permission should be configurable > --- > > Key: DRILL-5391 > URL: https://issues.apache.org/jira/browse/DRILL-5391 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0, 1.10.0 > Environment: CentOS 7, HDP 2.4 >Reporter: Chua Tianxiang >Priority: Minor > Attachments: Drill-1-10.PNG, Drill-1-9.PNG > > > In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on > Drill 1.10, the same commands creates a folder with permission 775. Both > drills are started with root user, installed on the same servers and accesses > the same HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable
[ https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945476#comment-15945476 ] Zelaine Fong commented on DRILL-5391: - Yes, I can see how default of 775/664 makes more sense, and from what you're saying, that matches Hive's defaults. But I'm still not clear as to why Drill previously was using a default of 777/666. You indicate it's the default dictated by the file system. So, does that mean that in the case of HDFS, the default is 777/666? > CTAS: folder and file permission should be configurable > --- > > Key: DRILL-5391 > URL: https://issues.apache.org/jira/browse/DRILL-5391 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0, 1.10.0 > Environment: CentOS 7, HDP 2.4 >Reporter: Chua Tianxiang >Priority: Minor > Attachments: Drill-1-10.PNG, Drill-1-9.PNG > > > In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on > Drill 1.10, the same commands creates a folder with permission 775. Both > drills are started with root user, installed on the same servers and accesses > the same HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable
[ https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945372#comment-15945372 ] Arina Ielchiieva commented on DRILL-5391: - For example, in HIVE newly created folder and files are dictated by hive.files.umask.value (which defaults to 002 -> with this mask default directory permissions are 775 and default file permissions are 664). > CTAS: folder and file permission should be configurable > --- > > Key: DRILL-5391 > URL: https://issues.apache.org/jira/browse/DRILL-5391 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0, 1.10.0 > Environment: CentOS 7, HDP 2.4 >Reporter: Chua Tianxiang >Priority: Minor > Attachments: Drill-1-10.PNG, Drill-1-9.PNG > > > In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on > Drill 1.10, the same commands creates a folder with permission 775. Both > drills are started with root user, installed on the same servers and accesses > the same HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable
[ https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945335#comment-15945335 ] Arina Ielchiieva commented on DRILL-5391: - During CTTAS implementation we needed to set permission during folder / file creation, and in code not to distinguish between for which tables folder / files are created, we pass StorageStrategy object which tells Drill folder / file permission that should be used. Before that changes Drill has used default permission dictated by underlined file system used for CTAS. In most cases and files systems, defaults are 775 & 644 for folder and files respectively. I didn't really make the permissions more restrictive, it's just approach has changed. > CTAS: folder and file permission should be configurable > --- > > Key: DRILL-5391 > URL: https://issues.apache.org/jira/browse/DRILL-5391 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0, 1.10.0 > Environment: CentOS 7, HDP 2.4 >Reporter: Chua Tianxiang >Priority: Minor > Attachments: Drill-1-10.PNG, Drill-1-9.PNG > > > In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on > Drill 1.10, the same commands creates a folder with permission 775. Both > drills are started with root user, installed on the same servers and accesses > the same HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5391) CTAS: folder and file permission should be configurable
[ https://issues.apache.org/jira/browse/DRILL-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945305#comment-15945305 ] Zelaine Fong commented on DRILL-5391: - [~arina] - was there a specific reason you decided to make the permissions more restrictive? > CTAS: folder and file permission should be configurable > --- > > Key: DRILL-5391 > URL: https://issues.apache.org/jira/browse/DRILL-5391 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.9.0, 1.10.0 > Environment: CentOS 7, HDP 2.4 >Reporter: Chua Tianxiang >Priority: Minor > Attachments: Drill-1-10.PNG, Drill-1-9.PNG > > > In Drill 1.9, CREATE TABLE AS creates a folder with permissions 777, while on > Drill 1.10, the same commands creates a folder with permission 775. Both > drills are started with root user, installed on the same servers and accesses > the same HDFS. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
