[ https://issues.apache.org/jira/browse/HIVE-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461999#comment-15461999 ]
Hive QA commented on HIVE-13989: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12812615/HIVE-13989.1.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1104/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1104/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1104/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]] + export JAVA_HOME=/usr/java/jdk1.8.0_25 + JAVA_HOME=/usr/java/jdk1.8.0_25 + export PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-1104/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at b74c4d0 HIVE-13383 : RetryingMetaStoreClient retries non retriable embedded metastore client (Thejas Nair via Ashutosh Chauhan) + git clean -f -d Removing ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java.orig Removing ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArrayByField.java Removing ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFSortArrayByField.java Removing ql/src/test/queries/clientnegative/udf_sort_array_by_wrong1.q Removing ql/src/test/queries/clientnegative/udf_sort_array_by_wrong2.q Removing ql/src/test/queries/clientnegative/udf_sort_array_by_wrong3.q Removing ql/src/test/queries/clientpositive/udf_sort_array_by.q Removing ql/src/test/results/clientnegative/udf_sort_array_by_wrong1.q.out Removing ql/src/test/results/clientnegative/udf_sort_array_by_wrong2.q.out Removing ql/src/test/results/clientnegative/udf_sort_array_by_wrong3.q.out Removing ql/src/test/results/clientpositive/udf_sort_array_by.q.out + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at b74c4d0 HIVE-13383 : RetryingMetaStoreClient retries non retriable embedded metastore client (Thejas Nair via Ashutosh Chauhan) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch error: patch failed: hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java:32 error: hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java: patch does not apply error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:2916 error: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: patch does not apply The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12812615 - PreCommit-HIVE-MASTER-Build > Extended ACLs are not handled according to specification > -------------------------------------------------------- > > Key: HIVE-13989 > URL: https://issues.apache.org/jira/browse/HIVE-13989 > Project: Hive > Issue Type: Bug > Components: HCatalog > Affects Versions: 1.2.1, 2.0.0 > Reporter: Chris Drome > Assignee: Chris Drome > Attachments: HIVE-13989-branch-1.patch, HIVE-13989.1-branch-1.patch, > HIVE-13989.1.patch > > > Hive takes two approaches to working with extended ACLs depending on whether > data is being produced via a Hive query or HCatalog APIs. A Hive query will > run an FsShell command to recursively set the extended ACLs for a directory > sub-tree. HCatalog APIs will attempt to build up the directory sub-tree > programmatically and runs some code to set the ACLs to match the parent > directory. > Some incorrect assumptions were made when implementing the extended ACLs > support. Refer to https://issues.apache.org/jira/browse/HDFS-4685 for the > design documents of extended ACLs in HDFS. These documents model the > implementation after the POSIX implementation on Linux, which can be found at > http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html. > The code for setting extended ACLs via HCatalog APIs is found in > HdfsUtils.java: > {code} > if (aclEnabled) { > aclStatus = sourceStatus.getAclStatus(); > if (aclStatus != null) { > LOG.trace(aclStatus.toString()); > aclEntries = aclStatus.getEntries(); > removeBaseAclEntries(aclEntries); > //the ACL api's also expect the tradition user/group/other permission > in the form of ACL > aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.USER, > sourcePerm.getUserAction())); > aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.GROUP, > sourcePerm.getGroupAction())); > aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.OTHER, > sourcePerm.getOtherAction())); > } > } > {code} > We found that DEFAULT extended ACL rules were not being inherited properly by > the directory sub-tree, so the above code is incomplete because it > effectively drops the DEFAULT rules. The second problem is with the call to > {{sourcePerm.getGroupAction()}}, which is incorrect in the case of extended > ACLs. When extended ACLs are used the GROUP permission is replaced with the > extended ACL mask. So the above code will apply the wrong permissions to the > GROUP. Instead the correct GROUP permissions now need to be pulled from the > AclEntry as returned by {{getAclStatus().getEntries()}}. See the > implementation of the new method {{getDefaultAclEntries}} for details. > Similar issues exist with the HCatalog API. None of the API accounts for > setting extended ACLs on the directory sub-tree. The changes to the HCatalog > API allow the extended ACLs to be passed into the required methods similar to > how basic permissions are passed in. When building the directory sub-tree the > extended ACLs of the table directory are inherited by all sub-directories, > including the DEFAULT rules. > Replicating the problem: > Create a table to write data into (I will use acl_test as the destination and > words_text as the source) and set the ACLs as follows: > {noformat} > $ hdfs dfs -setfacl -m > default:user::rwx,default:group::r-x,default:mask::rwx,default:user:hdfs:rwx,group::r-x,user:hdfs:rwx > /user/cdrome/hive/acl_test > $ hdfs dfs -ls -d /user/cdrome/hive/acl_test > drwxrwx---+ - cdrome hdfs 0 2016-07-13 20:36 > /user/cdrome/hive/acl_test > $ hdfs dfs -getfacl -R /user/cdrome/hive/acl_test > # file: /user/cdrome/hive/acl_test > # owner: cdrome > # group: hdfs > user::rwx > user:hdfs:rwx > group::r-x > mask::rwx > other::--- > default:user::rwx > default:user:hdfs:rwx > default:group::r-x > default:mask::rwx > default:other::--- > {noformat} > Note that the basic GROUP permission is set to {{rwx}} after setting the > ACLs. The ACLs explicitly set the DEFAULT rules and a rule specifically for > the {{hdfs}} user. > Run the following query to populate the table: > {noformat} > insert into acl_test partition (dt='a', ds='b') select a, b from words_text > where dt = 'c'; > {noformat} > Note that words_text only has a single partition key. > Now examine the ACLs for the resulting directories: > {noformat} > $ hdfs dfs -getfacl -R /user/cdrome/hive/acl_test > # file: /user/cdrome/hive/acl_test > # owner: cdrome > # group: hdfs > user::rwx > user:hdfs:rwx > group::r-x > mask::rwx > other::--- > default:user::rwx > default:user:hdfs:rwx > default:group::r-x > default:mask::rwx > default:other::--- > # file: /user/cdrome/hive/acl_test/dt=a > # owner: cdrome > # group: hdfs > user::rwx > user:hdfs:rwx > group::rwx > mask::rwx > other::--- > default:user::rwx > default:user:hdfs:rwx > default:group::rwx > default:mask::rwx > default:other::--- > # file: /user/cdrome/hive/acl_test/dt=a/ds=b > # owner: cdrome > # group: hdfs > user::rwx > user:hdfs:rwx > group::rwx > mask::rwx > other::--- > default:user::rwx > default:user:hdfs:rwx > default:group::rwx > default:mask::rwx > default:other::--- > # file: /user/cdrome/hive/acl_test/dt=a/ds=b/000000_0.deflate > # owner: cdrome > # group: hdfs > user::rwx > user:hdfs:rwx > group::rwx > mask::rwx > other::--- > {noformat} > Note that the GROUP permission is now erroneously set to {{rwx}} because of > the code mentioned above; it is set to the same value as the ACL mask. > The code changes for the HCatalog APIs is synonymous to the > {{applyGroupAndPerms}} method which ensures that all new directories are > created with the same permissions as the table. This patch will ensure that > changes to intermediate directories will not be propagated, instead the table > ACLs will be applied to all new directories created. > I would also like to call out that the older versions of HDFS which support > ACLs had a number issues in addition to those mentioned here which appear to > have been addressed in later versions of Hadoop. This patch was originally > written to work with a version of Hadoop-2.6, we are now using Hadoop-2.7 > which appears to have fixed some of them. However, I think that this patch is > still required for correct behavior of ACLs with Hive/HCatalog. -- This message was sent by Atlassian JIRA (v6.3.4#6332)