[jira] [Created] (HADOOP-14135) Remove URI parameter in AWSCredentialProvider constructors
Mingliang Liu created HADOOP-14135:
--
Summary: Remove URI parameter in AWSCredentialProvider constructors
Key: HADOOP-14135
URL: https://issues.apache.org/jira/browse/HADOOP-14135
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Mingliang Liu
Assignee: Mingliang Liu
This was from comment in [HADOOP-13252].
It looks like the URI parameter is not needed for our AWSCredentialProvider
constructors. This was useful because we relied on URI parameter for retrieving
user:pass. Now in binding URIs, we have
{code}
279 S3xLoginHelper.Login creds = getAWSAccessKeys(binding, conf);
280 credentials.add(new BasicAWSCredentialsProvider(
281 creds.getUser(), creds.getPassword()));
{code}
This way, we only need configuration object (if necessary) for all
AWSCredentialProvider implementations. The benefit is that, if we create
AWSCredentialProvider list for DynamoDB, we don't have to pass down the
associated file system URI. This might be useful to S3Guard tools.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/ [Feb 27, 2017 10:39:14 AM] (yqlin) HADOOP-14119. Remove unused imports from GzipCodec.java. Contributed by [Feb 27, 2017 10:46:37 AM] (aajisaka) MAPREDUCE-6841. Fix dead link in MapReduce tutorial document. [Feb 27, 2017 3:08:29 PM] (naganarasimha_gr) YARN-5703. ReservationAgents are not correctly configured. Contributed [Feb 28, 2017 1:07:45 AM] (wang) HDFS-11382. Persist Erasure Coding Policy ID in a new optional field in [Feb 28, 2017 4:42:13 AM] (yzhang) HADOOP-14127. Add log4j configuration to enable logging in [Feb 28, 2017 4:47:44 AM] (aajisaka) HADOOP-14118. move jets3t into a dependency on hadoop-aws JAR. [Feb 28, 2017 6:10:57 AM] (kasha) YARN-6231. FairSchedulerTestBase helper methods should call -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.qjournal.client.TestQuorumJournalManager hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker hadoop.fs.TestSymlinkHdfsFileContext hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation hadoop.yarn.server.nodemanager.containermanager.TestContainerManager hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestDiskFailures Timed out junit tests : org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/diff-compile-javac-root.txt [184K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/diff-patch-shellcheck.txt [24K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [420K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [36K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/331/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/ [Feb 27, 2017 3:08:29 PM] (naganarasimha_gr) YARN-5703. ReservationAgents are not correctly configured. Contributed [Feb 28, 2017 1:07:45 AM] (wang) HDFS-11382. Persist Erasure Coding Policy ID in a new optional field in [Feb 28, 2017 4:42:13 AM] (yzhang) HADOOP-14127. Add log4j configuration to enable logging in [Feb 28, 2017 4:47:44 AM] (aajisaka) HADOOP-14118. move jets3t into a dependency on hadoop-aws JAR. [Feb 28, 2017 6:10:57 AM] (kasha) YARN-6231. FairSchedulerTestBase helper methods should call [Feb 28, 2017 8:19:19 AM] (sunilg) YARN-5335. Use em-table in app/nodes pages for new YARN UI. Contributed [Feb 28, 2017 10:51:56 AM] (yqlin) YARN-6239. Fix javadoc warnings in YARN that caused by deprecated -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-compile-root.txt [140K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-compile-root.txt [140K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-compile-root.txt [140K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [472K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [68K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/243/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt [8.0K]
[jira] [Created] (HADOOP-14132) Filesystem discovery to stop loading implementation classes
Steve Loughran created HADOOP-14132: --- Summary: Filesystem discovery to stop loading implementation classes Key: HADOOP-14132 URL: https://issues.apache.org/jira/browse/HADOOP-14132 Project: Hadoop Common Issue Type: Improvement Components: fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift Affects Versions: 2.7.3 Reporter: Steve Loughran Assignee: Steve Loughran Integration testing of Hadoop with the HADOOP-14040 has shown up that the move to a shaded AWS JAR is slowing all hadoop client code down. I believe this is due to how we use service discovery to identify FS implementations: the implementation classes themselves are instantiated. This has known problems today with classloading, but clearly impacts performance too, especially with complex transitive dependencies unique to the loaded class. Proposed: have lightweight service declaration classes which implement an interface declaring # schema # classname of FileSystem impl # classname of AbstractFS impl # homepage (for third party code, support, etc) These are what we register and scan in the FS to look for services. This will leave the question about what to do for existing filesystems? I think we'll need to retain the old code for external ones, while moving the hadoop modules to the new ones -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
