[jira] [Reopened] (HADOOP-17725) Improve error message for token providers in ABFS
[
https://issues.apache.org/jira/browse/HADOOP-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mehakmeet Singh reopened HADOOP-17725:
--
Re-opening as it caused some unexpected failures in our environment. Since, MSI
tenant ID and client ID are optional, as stated in:
https://hadoop.apache.org/docs/stable/hadoop-azure/abfs.html#OAuth_2.0_Client_Credentials,
I think we should not mandate them to be set.
> Improve error message for token providers in ABFS
> -
>
> Key: HADOOP-17725
> URL: https://issues.apache.org/jira/browse/HADOOP-17725
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure, hadoop-thirdparty
>Affects Versions: 3.3.0
>Reporter: Ivan Sadikov
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> It would be good to improve error messages for token providers in ABFS.
> Currently, when a configuration key is not found or mistyped, the error is
> not very clear on what went wrong. It would be good to indicate that the key
> was required but not found in Hadoop configuration when creating a token
> provider.
> For example, when running the following code:
> {code:java}
> import org.apache.hadoop.conf._
> import org.apache.hadoop.fs._
> val conf = new Configuration()
> conf.set("fs.azure.account.auth.type", "OAuth")
> conf.set("fs.azure.account.oauth.provider.type",
> "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
> conf.set("fs.azure.account.oauth2.client.id", "my-client-id")
> //
> conf.set("fs.azure.account.oauth2.client.secret.my-account.dfs.core.windows.net",
> "my-secret")
> conf.set("fs.azure.account.oauth2.client.endpoint", "my-endpoint")
> val path = new Path("abfss://contai...@my-account.dfs.core.windows.net/")
> val fs = path.getFileSystem(conf)
> fs.getFileStatus(path){code}
> The following exception is thrown:
> {code:java}
> TokenAccessProviderException: Unable to load OAuth token provider class.
> ...
> Caused by: UncheckedExecutionException: java.lang.NullPointerException:
> clientSecret
> ...
> Caused by: NullPointerException: clientSecret {code}
> which does not tell what configuration key was not loaded.
>
> IMHO, it would be good if the exception was something like this:
> {code:java}
> TokenAccessProviderException: Unable to load OAuth token provider class.
> ...
> Caused by: ConfigurationPropertyNotFoundException: Configuration property
> fs.azure.account.oauth2.client.secret not found. {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/714/ [Dec 8, 2021 12:21:48 PM] (noreply) HDFS-16324. Fix error log in BlockManagerSafeMode (#3661) [Dec 8, 2021 12:22:40 PM] (noreply) HDFS-15788. Correct the statement for pmem cache to reflect cache persistence support (#3761) [Dec 8, 2021 3:03:42 PM] (Szilard Nemeth) YARN-11031. Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched. Contributed by Tamas Domok [Dec 8, 2021 3:41:35 PM] (Szilard Nemeth) YARN-11038. Fix testQueueSubmitWithACL* tests in TestAppManager. Contributed by Tamas Domok [Dec 8, 2021 6:02:10 PM] (noreply) YARN-11005. Implement the core QUEUE_LENGTH_THEN_RESOURCES OContainer allocation policy (#3717) [Dec 8, 2021 6:38:54 PM] (noreply) HADOOP-18034. Bump mina-core from 2.0.16 to 2.1.5 in /hadoop-project. (#3706) [Dec 9, 2021 4:24:09 AM] (noreply) HDFS-16333. fix balancer bug when transfer an EC block (#3679) [Dec 9, 2021 4:47:41 AM] (noreply) HDFS-16285. Make HDFS ownership tools cross platform (#3588) [Dec 9, 2021 7:46:10 AM] (noreply) HADOOP-18001. Upgrade jetty version to 9.4.44 (#3700). Contributed by Yuan Luo. -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.security.authentication.server.TestLdapAuthenticationHandler hadoop.security.authentication.server.TestMultiSchemeAuthenticationHandler hadoop.hdfs.TestViewDistributedFileSystemContract hadoop.hdfs.server.blockmanagement.TestBlockInfoStriped hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId hadoop.hdfs.server.balancer.TestBalancer hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.balancer.TestBalancerRPCDelay hadoop.hdfs.server.blockmanagement.TestReplicationPolicy hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes hadoop.hdfs.server.balancer.TestBalancerService hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand hadoop.hdfs.web.TestWebHdfsFileSystemContract hadoop.hdfs.server.datanode.TestBPOfferService hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics hadoop.tools.fedbalance.procedure.TestBalanceProcedureScheduler hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure hadoop.yarn.csi.client.TestCsiClient hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.TestDynamometerInfra cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/714/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/714/artifact/out/results-compile-javac-root.txt [360K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/714/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/714/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/714/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/714/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/714/artifact/out/results-pylint.txt [20K] shellcheck:
[jira] [Created] (HADOOP-18042) Fix jetty version in LICENSE-binary
Yuan Luo created HADOOP-18042: - Summary: Fix jetty version in LICENSE-binary Key: HADOOP-18042 URL: https://issues.apache.org/jira/browse/HADOOP-18042 Project: Hadoop Common Issue Type: Improvement Reporter: Yuan Luo -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/ [Dec 9, 2021 4:27:38 PM] (Akira Ajisaka) HADOOP-18035. Ignore unit test failures to run all the unit tests from root (#3765) [Dec 9, 2021 4:47:31 PM] (Akira Ajisaka) HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestTrash hadoop.fs.TestFileUtil hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/diff-compile-javac-root.txt [500K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/patch-mvnsite-root.txt [584K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/diff-patch-pylint.txt [48K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/diff-patch-shellcheck.txt [56K] shelldocs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/diff-patch-shelldocs.txt [48K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/patch-javadoc-root.txt [32K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [236K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [428K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [124K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/507/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K]
[jira] [Resolved] (HADOOP-17982) OpensslCipher initialization error should log a WARN message
[
https://issues.apache.org/jira/browse/HADOOP-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki resolved HADOOP-17982.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> OpensslCipher initialization error should log a WARN message
>
>
> Key: HADOOP-17982
> URL: https://issues.apache.org/jira/browse/HADOOP-17982
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms, security
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We spent months troubleshooting a RangerKMS performance problem, only to
> realize that the openssl library wasn't even loaded properly.
> The failure to load openssl lib is currently logged as a debug message during
> initialization. We really should upgrade it to at least INFO/WARN.
> {code}
> static {
> String loadingFailure = null;
> try {
> if (!NativeCodeLoader.buildSupportsOpenssl()) {
> PerformanceAdvisory.LOG.debug("Build does not support openssl");
> loadingFailure = "build does not support openssl.";
> } else {
> initIDs();
> }
> } catch (Throwable t) {
> loadingFailure = t.getMessage();
> LOG.debug("Failed to load OpenSSL Cipher.", t);
> } finally {
> loadingFailureReason = loadingFailure;
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
