Thanks, Philip! Since it's still unfixed, let me try to use another username.
在 2018-04-01 12:35:42，"Philip Zeyliger" <[email protected]> 写道： Hi Quanlong, You're running into https://issues.apache.org/jira/browse/HADOOP-7050. I found that JIRA via https://kb.informatica.com/solution/23/Pages/61/510035.aspx (and Google). What's surprising to me is that this code has been around in Hadoop for a long time. I think HiveServer2 must have changed in that it now invokes the impersonation code, whereas it must have not before. Here are some pointers: testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl: <name>hadoop.proxyuser.${USER}.hosts</name> https://github.com/apache/hadoop/blob/dc8e3432013153ac11d31d6b462aa96f8ca2c443/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/DefaultImpersonationProvider.java#L78 String usersGroupsRegEx = prefixRegEx + "[^.]*(" + Pattern.quote(CONF_USERS) + "|" + Pattern.quote(CONF_GROUPS) + ")"; To my eye, it looks like that regular expression is wrong and is what's disallowing usernames with periods/dots in them. (Once it's loosened, any code that's parsing these may also need to be fixed.) -- Philip On Fri, Mar 30, 2018 at 7:34 PM, Quanlong Huang <[email protected]> wrote: I failed to start the minicluster too but encountered another errors. HiveServer2 failed to launch and kept warning: 2018-03-30T18:54:05,526 WARN [HiveServer2-Handler-Pool: Thread-49] thrift.ThriftCLIService: Error opening session: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: quanlong.huang is not allowed to impersonate foo at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121] at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_121] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at com.sun.proxy.$Proxy37.open(Unknown Source) ~[?:?] at org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:411) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:363) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:189) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:423) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:312) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) [hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: quanlong.huang is not allowed to impersonate foo at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:596) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] ... 21 more Caused by: org.apache.hadoop.ipc.RemoteException: User: quanlong.huang is not allowed to impersonate foo at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1437) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1347) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at com.sun.proxy.$Proxy31.getFileInfo(Unknown Source) ~[?:?] at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:875) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at com.sun.proxy.$Proxy32.getFileInfo(Unknown Source) ~[?:?] at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1643) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1494) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1491) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?] at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1506) ~[hadoop-hdfs-client-3.0.0-cdh6.x-20180302.192732-2.jar:?] at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668) ~[hadoop-common-3.0.0-cdh6.x-20180302.191654-1.jar:?] at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:701) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:640) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:572) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539) ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT] ... 21 more I just rebase the codes and run ./buildall.sh -format Any thoughts? Thanks, Quanlong At 2018-03-30 13:08:33, "Dimitris Tsirogiannis" <[email protected]> wrote: >I enabled full logging in my postgres that hosts the sentry and metastore >db and I don't see the table being created. If anyone has gone through the >process, can you: a) verify that relation SENTRY_ROLE exists in your >sentry_policy db, and b) tell me how many relations are in your policy_db. > >Thanks >Dimitris > >On Thu, Mar 29, 2018 at 9:32 PM, Dimitris Tsirogiannis < >[email protected]> wrote: > >> Good point. I used -format that in theory handles both the metastore and >> the sentry policy dB. The sentry_policy db is created and has some tables >> but not the SENTRY_ROLE. >> >> Dimitris >> >> On Thu, Mar 29, 2018 at 6:29 PM Jim Apple <[email protected]> wrote: >> >>> I think I might have once fixed that using >>> >>> ./buildall.sh -notests -format_metastore -format_sentry_policy_db >>> >>> >>> On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis < >>> [email protected]> wrote: >>> >>> > I tried rebuilding my minicluster but Sentry refuses to start. I get >>> > "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does >>> that >>> > ring any bells? >>> > >>> > Thanks >>> > Dimitris >>> > >>> > On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <[email protected]> >>> > wrote: >>> > >>> > > Hi folks, >>> > > >>> > > I just sent off https://gerrit.cloudera.org/#/c/9743/ and >>> > > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which >>> changes >>> > > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry >>> 2.0, >>> > > and so on. This change *will not* be back-ported to 2.x. >>> > > >>> > > When you pull that change in, you'll need to re-build your minicluster >>> > > with, e.g., ./buildall.sh -testdata -format -notests. This will pull >>> in >>> > the >>> > > new dependencies, format your cluster, and load up all the data. As >>> you >>> > > know, it takes 1-2 hours. >>> > > >>> > > If you want to hold off, you can elso set export >>> > > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment. >>> > > >>> > > Note that this choice between versions happens at build time, and >>> CMake >>> > > depends on it. So, switching back and forth requires re-running CMake. >>> > > >>> > > Please let me know if you run into any trouble. This is a big enough >>> that >>> > > there may be some bumps on the road. >>> > > >>> > > -- Philip >>> > > >>> > >>> >>
