Pankaj Kumar created HBASE-18620:
------------------------------------

             Summary: Secure bulkload job fails when HDFS umask has limited 
scope
                 Key: HBASE-18620
                 URL: https://issues.apache.org/jira/browse/HBASE-18620
             Project: HBase
          Issue Type: Bug
          Components: security
            Reporter: Pankaj Kumar


By default "hbase.fs.tmp.dir" parameter value is 
/user/$\{user.name}/hbase-staging.

RegionServer creates the staging directory (hbase.bulkload.staging.dir, default 
value is hbase.fs.tmp.dir) during opening a region as below when 
SecureBulkLoadEndpoint configured in hbase.coprocessor.region.classes,
{noformat}
drwx------ - hbase hadoop 0 2017-08-12 13:55 /user/xyz
drwx--x--x - hbase hadoop 0 2017-08-12 13:55 /user/xyz/hbase-staging
drwx--x--x - hbase hadoop 0 2017-08-12 13:55 /user/xyz/hbase-staging/DONOTERASE
{noformat}

Here,
1. RegionServer is started using "xyz" linux user.
2. HDFS umask (fs.permissions.umask-mode) has been set as 077, so file/dir 
permission will not be wider than 700. "/user/xyz" directory (doesn't exist 
earlier) permission will be 700 and "/user/xyz/hbase-staging" will be 711 as we 
are just setting permission of staging directory not the parent directories 
which are created (fs.mkdirs()) by RegionServer.


Secure bulkload will fail as other user doesn't have EXECUTE permission on 
"/user/xyz" directory.


*Steps to reproduce:*
==================
1. Configure org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint in 
"hbase.coprocessor.region.classes" at client side.
2. Login to machine as "root" linux user.
3. kinit to any kerberos user except RegionServer kerberos user (say admin).
4. ImportTSV will create the user temp directory (hbase.fs.tmp.dir) while 
writing partition file, 
{noformat}
drwxrwxrwx - admin hadoop 0 2017-08-12 14:52 /user/root
drwxrwxrwx - admin hadoop 0 2017-08-12 14:52 /user/root/hbase-staging
{noformat}
4. During LoadIncrementalHFiles job,
- a. prepareBulkLoad() step - Random dir will be created by RegionServer 
credentials,
{noformat}
drwxrwxrwx - hbase hadoop 0 2017-08-12 14:58 
/user/xyz/hbase-staging/hbase__t1__e67b23m2ghe6fkn1bqrb95ak41ferj8957cdhsep4ebmpohm22nvi54vh8g3qh1
{noformat}

- b. secureBulkLoadHFiles() step - Family dir existence check and creation is 
done by using client user credentials. Here client operation will fail as below,
{noformat}
Caused by: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
 Permission denied: user=admin, access=EXECUTE, 
inode="/user/xyz/hbase-staging/admin__t1__e1f3m4r2prud9117thg5pdg91lkg0le0fdvtbbpg03epqg0f14lv54j8sqd8s0n6/cf1":hbase:hadoop:drwx------
        at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:342)
        at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:279)
        at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:223)
        at 
com.huawei.hadoop.adapter.hdfs.plugin.HWAccessControlEnforce.checkPermission(HWAccessControlEnforce.java:69)
{noformat}

So the root cause is "admin" user doesn't have EXECUTE permission over 
"/user/xyz", because RegionServer has created this intermediate parent 
directory during opening (SecureBulkLoadEndpoint) a region where the default 
permission is set as 700 based on the hdfs UMASK 077.


*Solution:*
=========
However it can be handled by the creating /user/xyz manually and setting 
sufficient permission explicitly. But we should handle this by setting 
sufficient permission to intermediate staging directories which is created by 
RegionServer.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to