[
https://issues.apache.org/jira/browse/HADOOP-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875158#comment-17875158
]
ASF GitHub Bot commented on HADOOP-19249:
-----------------------------------------
hadoop-yetus commented on PR #6984:
URL: https://github.com/apache/hadoop/pull/6984#issuecomment-2298657618
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 36s | | trunk passed |
| +1 :green_heart: | compile | 9m 3s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 8m 10s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 1m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 20s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 8m 31s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 8m 31s | | the patch passed |
| +1 :green_heart: | compile | 8m 17s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 8m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 42s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 1m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 46s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 16m 36s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 137m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6984/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6984 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux aa2cc8a8fca5 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 738c42ee1d1f654d12a3075a4e2771de2b5f2163 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6984/3/testReport/ |
| Max. process+thread count | 1274 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6984/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Getting NullPointerException when the unauthorised user tries to perform the
> key operation
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-19249
> URL: https://issues.apache.org/jira/browse/HADOOP-19249
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
> Reporter: Dhaval Shah
> Priority: Major
> Labels: pull-request-available
>
> While validating the tomcat 9.x in apache Ranger when user doesn't have
> appropriate permission in Ranger policies we faced the NPE for key operation
> using hadoop cmd.
> *Problem :*
> _Functionally -_ We are facing the NPE while performing key operations from
> hadoop cmd with the user not having permission in policy on cluster with
> tomcat v9.x. However with curl to Ranger KSM Server is working as expected.
> _Technically -_ Getting response message as null on client side in
> hadoop-common at
> [KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L565]
> *E.G.*
> _with Ranger KMS tomcat v9.x_
> {code:java}
> hadoop key list
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately if
> the provider requires a password and none is given.
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
> at
> org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
> at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}
> _on_ _Ranger KMS_ _tomcat v8.5.x_
> {code:java}
> hadoop key list
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately ifthe provider
> requires a password and none is given.
> Executing command failed with the following exception:
> AuthorizationException: User:xyzuser not allowed to do 'GET_KEYS'{code}
> *Debug logs on Ranger KMS Server side*
> 1.) Added logs in
> [KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java]
> in method _createResponse()_ and _toResponse()_ where we are generating
> response to send it to client i.e. _hadoop-common_
> Logs are exactly same on both the tomcat scenario. Refer below the added
> logs, detailed logs will be available in ranger kms log file on cluster.
> {code:java}
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== Entered into toResponse =========
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== exception
> =========org.apache.hadoop.security.authorize.AuthorizationException:
> User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== exception.getClass() =========class
> org.apache.hadoop.security.authorize.AuthorizationException
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== AuthorizationException =========
> 2024-07-25 11:35:51,452 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> [https-jsse-nio-9494-exec-2]: User syst...@root.comops.site (auth:KERBEROS)
> request GET
> https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names
> caused exception.
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ===== Entered into createResponse ======
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== status ======= Forbidden
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex =======
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex.getStackTrace() =======
> [Ljava.lang.StackTraceElement;@3e75ae9d
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex.getMessage() ======= User:systest
> not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex.toString() =======
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS' {code}
> 2.) Also added logs in
> [KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java]
> Adding logs in code base
> {code:java}
> public void setStatus(int sc, String sm) {
> LOG.info("========= setStatus with message============ ");
> statusCode = sc;
> msg = sm;
> LOG.info("========= sc ============ " +sc);
> LOG.info("========= msg ============ " +msg);
> if(sc == 403) {
> LOG.info("===== its 403 ====");
> super.setStatus(sc, sm);
> } else{
> super.setStatus(sc, sm);
> }
> } {code}
> LOGS:
> {code:java}
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ========= setStatus with message============
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ========= sc ============ 403
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ========= msg ============ Forbidden
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ===== its 403 ==== {code}
> This explains that the KMS server is sending the code and message
> appropriately.
> *Debug logs on Hadoop Common Client side*
> 1.) Added logs in
> [HttpExceptionUtils.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java]
> to make sure whether appropriate response is received.
> Logs will be available in ranger kms log file.
> {code:java}
> 2024-07-25 11:35:51,453 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ====== Entered into
> createJerseyExceptionResponse ====
> 2024-07-25 11:35:51,453 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== ex ========
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== ex.getMessage ========
> User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== status ======== Forbidden
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== status.getStatusCode ======== 403
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== status.getReasonPhrase ========
> Forbidden
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ======= response ========
> com.sun.jersey.core.spi.factory.ResponseImpl@5bd8a59b {code}
> 2.) Added logs exactly before NPE occurs i.e.
> [KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L564]
> Adding logs in code base
> LOG.info(" =========== conn ======== " + conn);
> Map<String, List<String>> map = conn.getHeaderFields();
> LOG.info("======= map ======== " + map);for (Map.Entry<String, List<String>>
> entry : map.entrySet()) {
> LOG.info("=============== " + "Key : " + entry.getKey() + " ,Value
> : " + entry.getValue());
> }
> LOG.info(" =========== conn.getResponseMessage ======== " +
> conn.getResponseMessage());
> LOG.info(" =========== conn.getResponseCode ======== " +
> conn.getResponseCode());if ((conn.getResponseCode() ==
> HttpURLConnection.HTTP_FORBIDDEN
> && (conn.getResponseMessage().equals(ANONYMOUS_REQUESTS_DISALLOWED) ||
> conn.getResponseMessage().contains(INVALID_SIGNATURE)))
> || conn.getResponseCode() == HttpURLConnection.HTTP_UNAUTHORIZED) {
> LOGS: This logs gets printed on terminal where we execute hadoop cmd .
> _with Ranger KMS tomcat v9.x_
> {code:java}
> hadoop key list
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ======== Entered into call
> ========
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =========== conn ========
> sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ======= map ========
> {Keep-Alive=[timeout=60], null=[HTTP/1.1 403],
> Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload],
> Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220],
> Date=[Thu, 25 Jul 2024 11:38:15 GMT], Content-Type=[application/json]}
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Keep-Alive ,Value : [timeout=60]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : null
> ,Value : [HTTP/1.1 403]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains;
> preload]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Server
> ,Value : [Apache Ranger]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Connection ,Value : [keep-alive]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Content-Length ,Value : [220]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Date
> ,Value : [Thu, 25 Jul 2024 11:38:15 GMT]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Content-Type ,Value : [application/json]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ===========
> conn.getResponseMessage ======== null
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ===========
> conn.getResponseCode ======== 403
> list [-provider <provider>] [-strict] [-metadata] [-help]:
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately if
> the provider requires a password and none is given.
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
> at
> org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
> at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}
> _with Ranger KMS tomcat v8.5.x_
> hadoop key list
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: ======== Entered into call
> ========
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =========== conn ========
> sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ds-tomcat-test1.root.comops.site:9494/kms/v1/keys/names24/07/25
> 11:02:25 INFO kms.KMSClientProvider: ======= map ========
> {Keep-Alive=[timeout=60], null=[HTTP/1.1 403 Forbidden],
> Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload],
> Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220],
> Date=[Thu, 25 Jul 2024 11:02:25 GMT], Content-Type=[application/json]}
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Keep-Alive ,Value : [timeout=60]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : null
> ,Value : [HTTP/1.1 403 Forbidden]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains;
> preload]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Server
> ,Value : [Apache Ranger]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Connection ,Value : [keep-alive]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Content-Length ,Value : [220]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Date
> ,Value : [Thu, 25 Jul 2024 11:02:25 GMT]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Content-Type ,Value : [application/json]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: ===========
> conn.getResponseMessage ======== Forbidden
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: ===========
> conn.getResponseCode ======== 403
> Cannot list keys for KeyProvider:
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@209da20d
> list [-provider <provider>] [-strict] [-metadata] [-help]:
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately ifthe provider
> requires a password and none is given.
> Executing command failed with the following exception:
> AuthorizationException: User:xyzuser not allowed to do 'GET_KEYS'
> Please notice
> _with tomcat v9.x : *Key : null ,Value : [HTTP/1.1 403]*_
> _with. tomcat v8.5.x : *Key : null ,Value : [HTTP/1.1 403 Forbidden]*_
> **
> Message "Forbidden" is not present with tomcat v9.x.
> It seems that tomcat v9.x is not setting the message and hadoop-common is
> trying to get where we are facing NPE.
> Also checked for _*org.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER*_ but
> its not available in tomcat 9.x
> Ref:
> Tomcat Doc for 8.5.x
> [https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/coyote/Constants.html#USE_CUSTOM_STATUS_MSG_IN_HEADER]
> Tomcat Doc for 9.x
> [https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/coyote/Constants.html]
> Thanks
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
