After reinstalling hadoop using "-Djava.io.tmpdir={other_tmp_folder} -D*jna*
.tmpdir={other_tmp_folder}" for both knoxcli.sh and gateway.sh, knoxcli.sh
worked:
[lianjia@prod1-namenode knox-server]$ sudo bin/knoxcli.sh user-auth-test
--cluster ui --u guest --p "{PASSWORD}" --d
LDAP authentication successful!
However, gateway.sh still not work:
[lianjia@prod1-namenode knox-server]$ curl -vvv -ik -u guest:"{PASSWORD}"
http://{knoxhost}:8443/gateway/ui/webhdfs/v1/user/?op=LISTSTATUS
2018-07-03 19:46:41,016 DEBUG hadoop.gateway
(GatewayFilter.java:doFilter(116)) - Received request: GET /webhdfs/v1/user/
2018-07-03 19:46:41,017 DEBUG authc.BasicHttpAuthenticationFilter
(BasicHttpAuthenticationFilter.java:createToken(308)) - Attempting to
execute login with headers [Basic Z3Vlc3Q6emhIQSVBQzIzKSg=]
2018-07-03 19:46:41,017 WARN authc.AbstractAuthenticator
(AbstractAuthenticator.java:authenticate(216)) - Authentication failed for
token submission [org.apache.shiro.authc.UsernamePasswordToken - guest,
rememberMe=false (10.0.21.129)]. Possible unexpected error? (Typical or
expected login exceptions should extend from AuthenticationException).
java.lang.NoClassDefFoundError: Could not initialize class
org.jvnet.libpam.impl.PAMLibrary$pam_conv
at org.jvnet.libpam.PAM.<init>(PAM.java:73)
at
org.apache.hadoop.gateway.shirorealm.KnoxPamRealm.doGetAuthenticationInfo(KnoxPamRealm.java:135)
at
org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)
at
org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)
at
org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)
at
org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)
at
org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)
at
org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)
at
org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
at
org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53)
at
org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:190)
at
org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)
at
org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)
at
org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)
at
org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)
at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at
org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at
org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at
org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at
org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at
org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
at
org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:50)
at
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
at
org.apache.hadoop.gateway.filter.XForwardedHeaderFilter.doFilter(XForwardedHeaderFilter.java:30)
at
org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
at
org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
at
org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
at
org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:139)
at
org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:91)
at
org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:141)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at
org.apache.hadoop.gateway.trace.TraceHandler.handle(TraceHandler.java:51)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at
org.apache.hadoop.gateway.filter.CorrelationHandler.handle(CorrelationHandler.java:39)
at
org.eclipse.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:529)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at
org.apache.hadoop.gateway.filter.PortMappingHelperHandler.handle(PortMappingHelperHandler.java:92)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at
org.eclipse.jetty.websocket.server.WebSocketHandler.handle(WebSocketHandler.java:112)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
2018-07-03 19:46:41,018 DEBUG servlet.SimpleCookie
(SimpleCookie.java:addCookieHeader(226)) - Added HttpServletResponse Cookie
[rememberMe=deleteMe; Path=/gateway/ui; Max-Age=0; Expires=Mon, 02-Jul-2018
19:46:41 GMT]
2018-07-03 19:46:41,018 DEBUG authc.BasicHttpAuthenticationFilter
(BasicHttpAuthenticationFilter.java:sendChallenge(274)) - Authentication
required: sending 401 Authentication challenge response
So we can narrow down the problem to authc.BasicHttpAuthenticationFilter.
Any idea?
.
On Tue, Jul 3, 2018 at 8:50 AM, Lian Jiang <[email protected]> wrote:
> Sandeep,
>
> I did not see the two folders that you mentioned. Instead, I enabled PAM
> debug by following https://chadmayfield.com/2016/06/15/enable-pam-debug-
> logging/. The only relevant info I found in pam log:
>
> 2018-07-03T15:44:04.574874+00:00 prod1-namenode java:
> pam_unix(login:auth): authentication failure; logname=lianjia uid=0 euid=0
> tty= ruser= rhost= user=guest
> 2018-07-03T15:44:04.575134+00:00 prod1-namenode audispd: node=localhost
> type=USER_AUTH msg=audit(1530632644.573:1275337): pid=24280 uid=0
> auid=5584 ses=397 msg='op=PAM:authentication grantors=? acct="guest"
> exe="/usr/jdk64/jdk1.8.0_112/bin/java" hostname=prod1-namenode.
> hdpad2.dwprodphx.oraclevcn.com addr=? terminal=pts/0 res=failed'
> 2018-07-03T15:44:06.412596+00:00 prod1-namenode audispd: node=localhost
> type=SYSCALL msg=audit(1530632646.410:1275338): arch=c000003e syscall=87
> success=no exit=-2 a0=7f88e1027ca0 a1=7f88e0549ad8 a2=7f88e1027ca0 a3=9
> items=1 ppid=24273 pid=24281 auid=5584 uid=0 gid=0 euid=0 suid=0 fsuid=0
> egid=0 sgid=0 fsgid=0 tty=pts0 ses=397 comm="java"
> exe="/usr/jdk64/jdk1.8.0_112/bin/java" key="delete-success"
>
> It does not help to me. Thanks for any clue.
>
>
>
> On Tue, Jul 3, 2018 at 7:51 AM, Sandeep Moré <[email protected]>
> wrote:
>
>> Hello Lian,
>> What errors do you see in the PAM logs (/var/log/secure,
>> /var/log/messages) for this login attempt ?
>>
>> Best,
>> Sandeep
>>
>> On Tue, Jul 3, 2018 at 1:25 AM Lian Jiang <[email protected]> wrote:
>>
>>> Thanks Larry.
>>>
>>> Setting "-Djava.io.tmpdir={other_tmp_folder} -D*jna*
>>> .tmpdir={other_tmp_folder}" in knoxcli.sh made it throw a different
>>> error.
>>>
>>> [lianjia@prod1-namenode knox-server]$ sudo bin/knoxcli.sh
>>> user-auth-test --cluster ui --u guest --p "{PASSWORD}" --d
>>> org.apache.shiro.authc.AuthenticationException:
>>> org.jvnet.libpam.PAMException: pam_authenticate failed : Authentication
>>> failure
>>> pam_authenticate failed : Authentication failure
>>> org.apache.shiro.authc.AuthenticationException:
>>> org.jvnet.libpam.PAMException: pam_authenticate failed : Authentication
>>> failure
>>> at org.apache.hadoop.gateway.shirorealm.KnoxPamRealm.handleAuth
>>> Failure(KnoxPamRealm.java:157)
>>> at org.apache.hadoop.gateway.shirorealm.KnoxPamRealm.doGetAuthe
>>> nticationInfo(KnoxPamRealm.java:137)
>>> at org.apache.shiro.realm.AuthenticatingRealm.getAuthentication
>>> Info(AuthenticatingRealm.java:568)
>>> at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingl
>>> eRealmAuthentication(ModularRealmAuthenticator.java:180)
>>> at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthe
>>> nticate(ModularRealmAuthenticator.java:267)
>>> at org.apache.shiro.authc.AbstractAuthenticator.authenticate(Ab
>>> stractAuthenticator.java:198)
>>> at org.apache.shiro.mgt.AuthenticatingSecurityManager.authentic
>>> ate(AuthenticatingSecurityManager.java:106)
>>> at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSec
>>> urityManager.java:270)
>>> at org.apache.shiro.subject.support.DelegatingSubject.login(
>>> DelegatingSubject.java:256)
>>> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand.authentic
>>> ateUser(KnoxCLI.java:1171)
>>> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand.authentic
>>> ateUser(KnoxCLI.java:1206)
>>> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPAuthCommand.execu
>>> te(KnoxCLI.java:1502)
>>> at org.apache.hadoop.gateway.util.KnoxCLI.run(KnoxCLI.java:143)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>>> at org.apache.hadoop.gateway.util.KnoxCLI.main(KnoxCLI.java:1777)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(
>>> Invoker.java:70)
>>> at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.
>>> java:39)
>>> at org.apache.hadoop.gateway.launcher.Command.run(Command.java:99)
>>> at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69)
>>> at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.
>>> java:46)
>>> Caused by: org.jvnet.libpam.PAMException: pam_authenticate failed :
>>> Authentication failure
>>> at org.jvnet.libpam.PAM.check(PAM.java:106)
>>> at org.jvnet.libpam.PAM.authenticate(PAM.java:124)
>>> at org.apache.hadoop.gateway.shirorealm.KnoxPamRealm.doGetAuthe
>>> nticationInfo(KnoxPamRealm.java:135)
>>> ... 22 more
>>> ERR: Unable to authenticate user: guest
>>>
>>>
>>>
>>>
>>> Looks like the /tmp error is gone. However, I found no clue about
>>> "Authentication
>>> failure" even pamtest works:
>>>
>>> [lianjia@prod1-namenode knox-server]$ sudo pamtester -v login guest
>>> authenticate
>>> pamtester: invoking pam_start(login, guest, ...)
>>> pamtester: performing operation - authenticate
>>> Password:
>>> pamtester: successfully authenticated
>>>
>>> Not sure how to go deeper. Still investigating. Any hint is highly
>>> appreciated.
>>>
>>> On Mon, Jul 2, 2018 at 12:32 PM, larry mccay <[email protected]>
>>> wrote:
>>>
>>>> Hi Lian -
>>>>
>>>> I haven't encountered this before. You will likely need to dig into the
>>>> shiro PAM support itself if not even lower into the Pam module code.
>>>>
>>>> I will try and find some time to dig a bit myself.
>>>>
>>>> Thanks,
>>>>
>>>> -larry
>>>>
>>>> On Mon, Jul 2, 2018, 2:58 PM Lian Jiang <[email protected]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> When /tmp has noexec, Knox OS auth throws error:
>>>>>
>>>>> [lianjia@prod1-namenode knox-server]$ sudo bin/knoxcli.sh
>>>>> user-auth-test --cluster ui --u guest --p "{PASSWORD}" --d
>>>>> org.apache.shiro.authc.AuthenticationException: Authentication failed
>>>>> for token submission [org.apache.shiro.authc.UsernamePasswordToken -
>>>>> guest, rememberMe=false]. Possible unexpected error? (Typical or expected
>>>>> login exceptions should extend from AuthenticationException).
>>>>> /tmp/jna-3506402/jna4211705767471308463.tmp:
>>>>> /tmp/jna-3506402/jna4211705767471308463.tmp: failed to map segment
>>>>> from shared object: Operation not permitted
>>>>> org.apache.shiro.authc.AuthenticationException: Authentication failed
>>>>> for token submission [org.apache.shiro.authc.UsernamePasswordToken -
>>>>> guest, rememberMe=false]. Possible unexpected error? (Typical or expected
>>>>> login exceptions should extend from AuthenticationException).
>>>>> at org.apache.shiro.authc.AbstractAuthenticator.authenticate(Ab
>>>>> stractAuthenticator.java:214)
>>>>> at org.apache.shiro.mgt.AuthenticatingSecurityManager.authentic
>>>>> ate(AuthenticatingSecurityManager.java:106)
>>>>> at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSec
>>>>> urityManager.java:270)
>>>>> at org.apache.shiro.subject.support.DelegatingSubject.login(
>>>>> DelegatingSubject.java:256)
>>>>> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand.authentic
>>>>> ateUser(KnoxCLI.java:1171)
>>>>> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand.authentic
>>>>> ateUser(KnoxCLI.java:1206)
>>>>> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPAuthCommand.execu
>>>>> te(KnoxCLI.java:1502)
>>>>> at org.apache.hadoop.gateway.util.KnoxCLI.run(KnoxCLI.java:143)
>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>>>>> at org.apache.hadoop.gateway.util.KnoxCLI.main(KnoxCLI.java:1777)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>>> ssorImpl.java:62)
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>>> thodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>> at org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(
>>>>> Invoker.java:70)
>>>>> at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.
>>>>> java:39)
>>>>> at org.apache.hadoop.gateway.launcher.Command.run(Command.java:99)
>>>>> at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.
>>>>> java:69)
>>>>> at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.
>>>>> java:46)
>>>>> Caused by: java.lang.UnsatisfiedLinkError:
>>>>> /tmp/jna-3506402/jna4211705767471308463.tmp:
>>>>> /tmp/jna-3506402/jna4211705767471308463.tmp: failed to map segment
>>>>> from shared object: Operation not permitted
>>>>> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
>>>>> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
>>>>> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
>>>>> at java.lang.Runtime.load0(Runtime.java:809)
>>>>> at java.lang.System.load(System.java:1086)
>>>>> at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(
>>>>> Native.java:761)
>>>>> at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:736)
>>>>> at com.sun.jna.Native.<clinit>(Native.java:131)
>>>>> at com.sun.jna.Pointer.<clinit>(Pointer.java:41)
>>>>> at com.sun.jna.Structure.<clinit>(Structure.java:1949)
>>>>> at org.jvnet.libpam.PAM.<init>(PAM.java:73)
>>>>> at org.apache.hadoop.gateway.shirorealm.KnoxPamRealm.doGetAuthe
>>>>> nticationInfo(KnoxPamRealm.java:135)
>>>>> at org.apache.shiro.realm.AuthenticatingRealm.getAuthentication
>>>>> Info(AuthenticatingRealm.java:568)
>>>>> at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingl
>>>>> eRealmAuthentication(ModularRealmAuthenticator.java:180)
>>>>> at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthe
>>>>> nticate(ModularRealmAuthenticator.java:267)
>>>>> at org.apache.shiro.authc.AbstractAuthenticator.authenticate(Ab
>>>>> stractAuthenticator.java:198)
>>>>> ... 18 more
>>>>> ERR: Unable to authenticate user: guest
>>>>>
>>>>> Setting "-Djava.io.tmpdir={other_tmp_folder} -D*jna*
>>>>> .tmpdir={other_tmp_folder}" in gateway.sh did not help.
>>>>>
>>>>> I cannot remove noexec for /tmp since it is required for our
>>>>> production. Any idea how to solve this issue? Thanks!
>>>>>
>>>>
>>>
>
