GitHub user tanxinz opened a pull request:
https://github.com/apache/spark/pull/16004
[STS][SQL]multiple STS running in the same machine throws
security.AccessControlException
## What changes were proposed in this pull request?
multiple Spark Thrift Servers running in the same machine throws
org.apache.hadoop.security.AccessControlException
(Please fill in changes proposed in this fix)
## How was this patch tested?
In spark2.0.2 , I have two users(**etl** , **dev** ) start Spark Thrift
Server in the same machine . I connected by beeline **etl** STS to execute a
command,and throwed org.apache.hadoop.security.AccessControlException.I don't
know why is **dev** user perform,not **etl**.
```
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
Permission denied: user=dev, access=EXECUTE,
inode="/user/hive/warehouse/tb_spark_sts/etl_cycle_id=20161122":etl:supergroup:drwxr-x---,group:etl:rwx,group:oth_dev:rwx,default:user:data_mining:r-x,default:group::rwx,default:group:etl:rwx,default:group:oth_dev:rwx,default:mask::rwx,default:other::---
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkAccessAcl(DefaultAuthorizationProvider.java:335)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:231)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:178)
at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6250)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3942)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:811)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getFileInfo(AuthorizationProviderProxyClientProtocol.java:502)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:815)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
```
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/spark branch-2.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16004.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16004
commit 39d2fdb51233ed9b1aaf3adaa3267853f5e58c0f
Author: frreiss
Date: 2016-11-02T06:00:17Z
[SPARK-17475][STREAMING] Delete CRC files if the filesystem doesn't use
checksum files
## What changes were proposed in this pull request?
When the metadata logs for various parts of Structured Streaming are stored
on non-HDFS filesystems such as NFS or ext4, the HDFSMetadataLog class leaves
hidden HDFS-style checksum (CRC) files in the log directory, one file per
batch. This PR modifies HDFSMetadataLog so that it detects the use of a
filesystem that doesn't use CRC files and removes the CRC files.
## How was this patch tested?
Modified an existing test case in HDFSMetadataLogSuite to check whether
HDFSMetadataLog correctly removes CRC files on the local POSIX filesystem. Ran
the entire regression suite.
Author: frreiss
Closes #15027 from frreiss/fred-17475.
(cherry picked from commit 620da3b4828b358
