Sandeep, I did not see the two folders that you mentioned. Instead, I enabled PAM debug by following https://chadmayfield.com/2016/06/15/enable-pam-debug-logging/. The only relevant info I found in pam log:
2018-07-03T15:44:04.574874+00:00 prod1-namenode java: pam_unix(login:auth): authentication failure; logname=lianjia uid=0 euid=0 tty= ruser= rhost= user=guest 2018-07-03T15:44:04.575134+00:00 prod1-namenode audispd: node=localhost type=USER_AUTH msg=audit(1530632644.573:1275337): pid=24280 uid=0 auid=5584 ses=397 msg='op=PAM:authentication grantors=? acct="guest" exe="/usr/jdk64/jdk1.8.0_112/bin/java" hostname= prod1-namenode.hdpad2.dwprodphx.oraclevcn.com addr=? terminal=pts/0 res=failed' 2018-07-03T15:44:06.412596+00:00 prod1-namenode audispd: node=localhost type=SYSCALL msg=audit(1530632646.410:1275338): arch=c000003e syscall=87 success=no exit=-2 a0=7f88e1027ca0 a1=7f88e0549ad8 a2=7f88e1027ca0 a3=9 items=1 ppid=24273 pid=24281 auid=5584 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=397 comm="java" exe="/usr/jdk64/jdk1.8.0_112/bin/java" key="delete-success" It does not help to me. Thanks for any clue. On Tue, Jul 3, 2018 at 7:51 AM, Sandeep Moré <[email protected]> wrote: > Hello Lian, > What errors do you see in the PAM logs (/var/log/secure, > /var/log/messages) for this login attempt ? > > Best, > Sandeep > > On Tue, Jul 3, 2018 at 1:25 AM Lian Jiang <[email protected]> wrote: > >> Thanks Larry. >> >> Setting "-Djava.io.tmpdir={other_tmp_folder} >> -D*jna*.tmpdir={other_tmp_folder}" >> in knoxcli.sh made it throw a different error. >> >> [lianjia@prod1-namenode knox-server]$ sudo bin/knoxcli.sh user-auth-test >> --cluster ui --u guest --p "{PASSWORD}" --d >> org.apache.shiro.authc.AuthenticationException: >> org.jvnet.libpam.PAMException: pam_authenticate failed : Authentication >> failure >> pam_authenticate failed : Authentication failure >> org.apache.shiro.authc.AuthenticationException: >> org.jvnet.libpam.PAMException: pam_authenticate failed : Authentication >> failure >> at org.apache.hadoop.gateway.shirorealm.KnoxPamRealm. >> handleAuthFailure(KnoxPamRealm.java:157) >> at org.apache.hadoop.gateway.shirorealm.KnoxPamRealm. >> doGetAuthenticationInfo(KnoxPamRealm.java:137) >> at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo( >> AuthenticatingRealm.java:568) >> at org.apache.shiro.authc.pam.ModularRealmAuthenticator. >> doSingleRealmAuthentication(ModularRealmAuthenticator.java:180) >> at org.apache.shiro.authc.pam.ModularRealmAuthenticator. >> doAuthenticate(ModularRealmAuthenticator.java:267) >> at org.apache.shiro.authc.AbstractAuthenticator.authenticate( >> AbstractAuthenticator.java:198) >> at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate( >> AuthenticatingSecurityManager.java:106) >> at org.apache.shiro.mgt.DefaultSecurityManager.login( >> DefaultSecurityManager.java:270) >> at org.apache.shiro.subject.support.DelegatingSubject. >> login(DelegatingSubject.java:256) >> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand. >> authenticateUser(KnoxCLI.java:1171) >> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand. >> authenticateUser(KnoxCLI.java:1206) >> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPAuthCommand. >> execute(KnoxCLI.java:1502) >> at org.apache.hadoop.gateway.util.KnoxCLI.run(KnoxCLI.java:143) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) >> at org.apache.hadoop.gateway.util.KnoxCLI.main(KnoxCLI.java:1777) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke( >> NativeMethodAccessorImpl.java:62) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke( >> DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at org.apache.hadoop.gateway.launcher.Invoker. >> invokeMainMethod(Invoker.java:70) >> at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39) >> at org.apache.hadoop.gateway.launcher.Command.run(Command.java:99) >> at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69) >> at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46) >> Caused by: org.jvnet.libpam.PAMException: pam_authenticate failed : >> Authentication failure >> at org.jvnet.libpam.PAM.check(PAM.java:106) >> at org.jvnet.libpam.PAM.authenticate(PAM.java:124) >> at org.apache.hadoop.gateway.shirorealm.KnoxPamRealm. >> doGetAuthenticationInfo(KnoxPamRealm.java:135) >> ... 22 more >> ERR: Unable to authenticate user: guest >> >> >> >> >> Looks like the /tmp error is gone. However, I found no clue about >> "Authentication >> failure" even pamtest works: >> >> [lianjia@prod1-namenode knox-server]$ sudo pamtester -v login guest >> authenticate >> pamtester: invoking pam_start(login, guest, ...) >> pamtester: performing operation - authenticate >> Password: >> pamtester: successfully authenticated >> >> Not sure how to go deeper. Still investigating. Any hint is highly >> appreciated. >> >> On Mon, Jul 2, 2018 at 12:32 PM, larry mccay <[email protected]> >> wrote: >> >>> Hi Lian - >>> >>> I haven't encountered this before. You will likely need to dig into the >>> shiro PAM support itself if not even lower into the Pam module code. >>> >>> I will try and find some time to dig a bit myself. >>> >>> Thanks, >>> >>> -larry >>> >>> On Mon, Jul 2, 2018, 2:58 PM Lian Jiang <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> When /tmp has noexec, Knox OS auth throws error: >>>> >>>> [lianjia@prod1-namenode knox-server]$ sudo bin/knoxcli.sh >>>> user-auth-test --cluster ui --u guest --p "{PASSWORD}" --d >>>> org.apache.shiro.authc.AuthenticationException: Authentication failed >>>> for token submission [org.apache.shiro.authc.UsernamePasswordToken - >>>> guest, rememberMe=false]. Possible unexpected error? (Typical or expected >>>> login exceptions should extend from AuthenticationException). >>>> /tmp/jna-3506402/jna4211705767471308463.tmp: >>>> /tmp/jna-3506402/jna4211705767471308463.tmp: >>>> failed to map segment from shared object: Operation not permitted >>>> org.apache.shiro.authc.AuthenticationException: Authentication failed >>>> for token submission [org.apache.shiro.authc.UsernamePasswordToken - >>>> guest, rememberMe=false]. Possible unexpected error? (Typical or expected >>>> login exceptions should extend from AuthenticationException). >>>> at org.apache.shiro.authc.AbstractAuthenticator.authenticate( >>>> AbstractAuthenticator.java:214) >>>> at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate( >>>> AuthenticatingSecurityManager.java:106) >>>> at org.apache.shiro.mgt.DefaultSecurityManager.login( >>>> DefaultSecurityManager.java:270) >>>> at org.apache.shiro.subject.support.DelegatingSubject. >>>> login(DelegatingSubject.java:256) >>>> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand. >>>> authenticateUser(KnoxCLI.java:1171) >>>> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand. >>>> authenticateUser(KnoxCLI.java:1206) >>>> at org.apache.hadoop.gateway.util.KnoxCLI$LDAPAuthCommand. >>>> execute(KnoxCLI.java:1502) >>>> at org.apache.hadoop.gateway.util.KnoxCLI.run(KnoxCLI.java:143) >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) >>>> at org.apache.hadoop.gateway.util.KnoxCLI.main(KnoxCLI.java:1777) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke( >>>> NativeMethodAccessorImpl.java:62) >>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke( >>>> DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>> at org.apache.hadoop.gateway.launcher.Invoker. >>>> invokeMainMethod(Invoker.java:70) >>>> at org.apache.hadoop.gateway.launcher.Invoker.invoke( >>>> Invoker.java:39) >>>> at org.apache.hadoop.gateway.launcher.Command.run(Command.java:99) >>>> at org.apache.hadoop.gateway.launcher.Launcher.run( >>>> Launcher.java:69) >>>> at org.apache.hadoop.gateway.launcher.Launcher.main( >>>> Launcher.java:46) >>>> Caused by: java.lang.UnsatisfiedLinkError: >>>> /tmp/jna-3506402/jna4211705767471308463.tmp: >>>> /tmp/jna-3506402/jna4211705767471308463.tmp: failed to map segment >>>> from shared object: Operation not permitted >>>> at java.lang.ClassLoader$NativeLibrary.load(Native Method) >>>> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) >>>> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) >>>> at java.lang.Runtime.load0(Runtime.java:809) >>>> at java.lang.System.load(System.java:1086) >>>> at com.sun.jna.Native.loadNativeDispatchLibraryFromC >>>> lasspath(Native.java:761) >>>> at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:736) >>>> at com.sun.jna.Native.<clinit>(Native.java:131) >>>> at com.sun.jna.Pointer.<clinit>(Pointer.java:41) >>>> at com.sun.jna.Structure.<clinit>(Structure.java:1949) >>>> at org.jvnet.libpam.PAM.<init>(PAM.java:73) >>>> at org.apache.hadoop.gateway.shirorealm.KnoxPamRealm. >>>> doGetAuthenticationInfo(KnoxPamRealm.java:135) >>>> at org.apache.shiro.realm.AuthenticatingRealm. >>>> getAuthenticationInfo(AuthenticatingRealm.java:568) >>>> at org.apache.shiro.authc.pam.ModularRealmAuthenticator. >>>> doSingleRealmAuthentication(ModularRealmAuthenticator.java:180) >>>> at org.apache.shiro.authc.pam.ModularRealmAuthenticator. >>>> doAuthenticate(ModularRealmAuthenticator.java:267) >>>> at org.apache.shiro.authc.AbstractAuthenticator.authenticate( >>>> AbstractAuthenticator.java:198) >>>> ... 18 more >>>> ERR: Unable to authenticate user: guest >>>> >>>> Setting "-Djava.io.tmpdir={other_tmp_folder} >>>> -D*jna*.tmpdir={other_tmp_folder}" >>>> in gateway.sh did not help. >>>> >>>> I cannot remove noexec for /tmp since it is required for our >>>> production. Any idea how to solve this issue? Thanks! >>>> >>> >>
