[jira] [Created] (HADOOP-14299) Hadoop Renew Thread for proxy users
Prabhu Joseph created HADOOP-14299: -- Summary: Hadoop Renew Thread for proxy users Key: HADOOP-14299 URL: https://issues.apache.org/jira/browse/HADOOP-14299 Project: Hadoop Common Issue Type: Bug Components: hdfs-client Affects Versions: 2.7.1 Reporter: Prabhu Joseph Currently Hadoop Client has a separate renew thread which is created only for Authentication type Kerberos and not for Proxy. So for proxy users, a yarn client monitoring a long running job will fail after initial ticket lifetime with GSS initiate failed unless there is a manual re-kinit. https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1030 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14300) GraphiteSink reports metrics containing undesirable whitespaces
Damien Claveau created HADOOP-14300:
---
Summary: GraphiteSink reports metrics containing undesirable
whitespaces
Key: HADOOP-14300
URL: https://issues.apache.org/jira/browse/HADOOP-14300
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Affects Versions: 2.7.3, 2.7.2, 2.7.1, 2.7.0, 2.6.0, 2.5.0
Reporter: Damien Claveau
Priority: Trivial
According to the Graphite documentation, the data sent must be in the following
format:
But some tag values that are flattened in the metricpath string sometimes
include whitespaces and violate the carbon/graphite format.
For example, the dirpath enumeration from the datanode :
FSDatasetState.org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.Context=FSDatasetState.StorageInfo=FSDataset{dirpath='[/data/1/hdfs/data/current,
/data/2/hdfs/data/current]'}.Hostname=worker1.Capacity 78436466688 1491986609
Currently, none of the aggregators available in the Graphite ecosystem can
handle these malformed metrics. These whitespaces should be replaced by
underscores.
I will be happy to attach a small patch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14301) Deprecate SharedInstanceProfileCredentialsProvider in branch-2.
Chris Nauroth created HADOOP-14301:
--
Summary: Deprecate SharedInstanceProfileCredentialsProvider in
branch-2.
Key: HADOOP-14301
URL: https://issues.apache.org/jira/browse/HADOOP-14301
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
[HADOOP-13727] added the {{SharedInstanceProfileCredentialsProvider}}, which
effectively reduces high number of connections to EC2 Instance Metadata Service
caused by {{InstanceProfileCredentialsProvider}}. That patch, in order to
prevent the throttling problem, defined new class
{{SharedInstanceProfileCredentialsProvider}} as a subclass of
{{InstanceProfileCredentialsProvider}}, which enforces creation of only a
single instance.
Per [HADOOP-13050], we upgraded the AWS Java SDK. Since then, the
{{InstanceProfileCredentialsProvider}} in SDK code internally enforces a
singleton. {{SharedInstanceProfileCredentialsProvider}} can be deprecated as of
release 2.9.0.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14301) Deprecate SharedInstanceProfileCredentialsProvider in branch-2.
[
https://issues.apache.org/jira/browse/HADOOP-14301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HADOOP-14301.
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
+1 and committed to branch-2. [~liuml07], thank you for the patch.
> Deprecate SharedInstanceProfileCredentialsProvider in branch-2.
> ---
>
> Key: HADOOP-14301
> URL: https://issues.apache.org/jira/browse/HADOOP-14301
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.9.0
>
> Attachments: HADOOP-14248-branch-2.002.patch
>
>
> [HADOOP-13727] added the {{SharedInstanceProfileCredentialsProvider}}, which
> effectively reduces high number of connections to EC2 Instance Metadata
> Service caused by {{InstanceProfileCredentialsProvider}}. That patch, in
> order to prevent the throttling problem, defined new class
> {{SharedInstanceProfileCredentialsProvider}} as a subclass of
> {{InstanceProfileCredentialsProvider}}, which enforces creation of only a
> single instance.
> Per [HADOOP-13050], we upgraded the AWS Java SDK. Since then, the
> {{InstanceProfileCredentialsProvider}} in SDK code internally enforces a
> singleton. {{SharedInstanceProfileCredentialsProvider}} can be deprecated as
> of release 2.9.0.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/ [Apr 11, 2017 1:44:18 PM] (jlowe) YARN-6195. Export UsedCapacity and AbsoluteUsedCapacity to JMX. [Apr 11, 2017 2:16:11 PM] (jlowe) YARN-6461. TestRMAdminCLI has very low test timeouts. Contributed by [Apr 11, 2017 2:32:13 PM] (weichiu) HADOOP-13665. Erasure Coding codec should support fallback coder. [Apr 11, 2017 5:12:08 PM] (naganarasimha_gr) YARN-6463. Correct spelling mistake in FileSystemRMStateStore. [Apr 11, 2017 9:56:18 PM] (wangda) YARN-6439. Fix ReservationSystem creation of default ReservationQueue. [Apr 11, 2017 10:29:38 PM] (liuml07) HDFS-11558. BPServiceActor thread name is too long. Contributed by [Apr 12, 2017 4:36:24 AM] (arp) HDFS-11630. TestThrottledAsyncCheckerTimeout fails intermittently in [Apr 12, 2017 5:03:09 AM] (cnauroth) HDFS-11163. Mover should move the file blocks to default storage once [Apr 12, 2017 6:31:47 AM] (sunilg) YARN-6421. Upgrade frontend-maven-plugin to 1.1 to fix new YARN UI build -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.mapred.TestShuffleHandler hadoop.tools.TestHadoopArchiveLogsRunner hadoop.metrics2.impl.TestKafkaMetrics hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.yarn.client.api.impl.TestAMRMClient hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore Timed out junit tests : org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache mvninstall: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-mvninstall-root.txt [492K] compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-compile-root.txt [20K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-compile-root.txt [20K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-compile-root.txt [20K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-unit-hadoop-assemblies.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [336K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [40K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-unit-hadoop-tools_hadoop-archive-logs.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-unit-hadoop-tools_hadoop-kafka.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/286/artifact/out/p
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/ [Apr 11, 2017 11:40:09 AM] (yqlin) HDFS-11637. Fix javac warning caused by the deprecated key used in [Apr 11, 2017 1:44:18 PM] (jlowe) YARN-6195. Export UsedCapacity and AbsoluteUsedCapacity to JMX. [Apr 11, 2017 2:16:11 PM] (jlowe) YARN-6461. TestRMAdminCLI has very low test timeouts. Contributed by [Apr 11, 2017 2:32:13 PM] (weichiu) HADOOP-13665. Erasure Coding codec should support fallback coder. [Apr 11, 2017 5:12:08 PM] (naganarasimha_gr) YARN-6463. Correct spelling mistake in FileSystemRMStateStore. [Apr 11, 2017 9:56:18 PM] (wangda) YARN-6439. Fix ReservationSystem creation of default ReservationQueue. [Apr 11, 2017 10:29:38 PM] (liuml07) HDFS-11558. BPServiceActor thread name is too long. Contributed by [Apr 12, 2017 4:36:24 AM] (arp) HDFS-11630. TestThrottledAsyncCheckerTimeout fails intermittently in [Apr 12, 2017 5:03:09 AM] (cnauroth) HDFS-11163. Mover should move the file blocks to default storage once [Apr 12, 2017 6:31:47 AM] (sunilg) YARN-6421. Upgrade frontend-maven-plugin to 1.1 to fix new YARN UI build -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes hadoop.hdfs.server.datanode.TestDataNodeUUID hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestDiskFailures hadoop.mapred.TestMRTimelineEventHandling hadoop.tools.TestDistCpSystem hadoop.tools.TestHadoopArchiveLogsRunner hadoop.metrics2.impl.TestKafkaMetrics Timed out junit tests : org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/diff-compile-javac-root.txt [184K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/whitespace-tabs.txt [1.2M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [372K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [60K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [88K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/patch-unit-hadoop-tools_hadoop-archive-logs.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/patch-unit-hadoop-tools_hadoop-kafka.txt [8.0K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/374/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org - To unsubscribe,
[jira] [Resolved] (HADOOP-14165) Add S3Guard.dirListingUnion in S3AFileSystem#listFiles, listLocatedStatus
[
https://issues.apache.org/jira/browse/HADOOP-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu resolved HADOOP-14165.
Resolution: Duplicate
Resolving as duplicate to [HADOOP-13926] and [HADOOP-14266].
> Add S3Guard.dirListingUnion in S3AFileSystem#listFiles, listLocatedStatus
> -
>
> Key: HADOOP-14165
> URL: https://issues.apache.org/jira/browse/HADOOP-14165
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Rajesh Balamohan
>Priority: Minor
>
> {{S3Guard::dirListingUnion}} merges information from backing store and DDB to
> create consistent view. This needs to be added in
> {{S3AFileSystem::listFiles}} and {{S3AFileSystem::listLocatedStatus}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
