[jira] [Created] (YARN-5919) FairSharePolicy is not reasonable when sorting the application
zhengchenyu created YARN-5919: - Summary: FairSharePolicy is not reasonable when sorting the application Key: YARN-5919 URL: https://issues.apache.org/jira/browse/YARN-5919 Project: Hadoop YARN Issue Type: Improvement Components: yarn Affects Versions: 2.7.1 Environment: CentOS-7.1 Reporter: zhengchenyu Priority: Critical Fix For: 2.7.1 I use the FairSharePolicy to sort the runnableApps in FSLeafQueue. For the comparation of FSAppAttempt, s1Needy and s1Needy must be negtive. So FairSharePolicy will use useToWeightRatio to compare. This algorithm only consider the ResourceUsage of the FSAppAttempt. There is a unreasonable phenomenon: A low ResourceUsage FsAppAttemp with no demand(or request) will be in front of the queue, then skip this chance of assigning container. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5920) TestRMHA.testTransitionedToStandbyShouldNotHang is flakey
Rohith Sharma K S created YARN-5920: --- Summary: TestRMHA.testTransitionedToStandbyShouldNotHang is flakey Key: YARN-5920 URL: https://issues.apache.org/jira/browse/YARN-5920 Project: Hadoop YARN Issue Type: Bug Components: test Reporter: Rohith Sharma K S In build [linkg|https://builds.apache.org/job/PreCommit-YARN-Build/13986/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt] test case timed out. This need to be investigated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5827) DockerLinuxContainer can't limit the cpu usage
[ https://issues.apache.org/jira/browse/YARN-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Kumpf resolved YARN-5827. --- Resolution: Duplicate Target Version/s: (was: 2.8.0) > DockerLinuxContainer can't limit the cpu usage > -- > > Key: YARN-5827 > URL: https://issues.apache.org/jira/browse/YARN-5827 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.0 > Environment: CentOS7.1 >Reporter: zhengchenyu >Assignee: Shane Kumpf > Original Estimate: 240h > Remaining Estimate: 240h > > When I use DockerLinuxContainer, I found that the function of limiting cpu > usage doesn't make effect. Because DockerLinuxContainer regard the exitCode > of "docker inpsect" as pid. But When I got this pid, the command of > "launch.sh" has started, then we can't add the child process or threads into > the cgroup, so the restrictions of cpu usage doesn't make effect。 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/ [Nov 21, 2016 2:25:11 AM] (aajisaka) HADOOP-13814. Sample configuration of KMS HTTP Authentication signature [Nov 21, 2016 5:05:15 AM] (aajisaka) HDFS-3. Document dfs.client.read.striped configuration in [Nov 21, 2016 5:28:34 AM] (brahma) HDFS-11144. TestFileCreationDelete#testFileCreationDeleteParent fails -1 overall The following subsystems voted -1: asflicense findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.TestRollingUpgrade hadoop.yarn.server.TestDiskFailures hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/diff-compile-javac-root.txt [164K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/diff-patch-shellcheck.txt [28K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/whitespace-tabs.txt [1.3M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [4.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [152K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [316K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/232/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/162/ [Nov 21, 2016 2:25:11 AM] (aajisaka) HADOOP-13814. Sample configuration of KMS HTTP Authentication signature [Nov 21, 2016 5:05:15 AM] (aajisaka) HDFS-3. Document dfs.client.read.striped configuration in [Nov 21, 2016 5:28:34 AM] (brahma) HDFS-11144. TestFileCreationDelete#testFileCreationDeleteParent fails -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.net.TestClusterTopology hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/162/artifact/out/patch-compile-root.txt [168K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/162/artifact/out/patch-compile-root.txt [168K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/162/artifact/out/patch-compile-root.txt [168K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/162/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [124K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/162/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [200K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/162/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/162/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/162/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt [20K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/162/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [88K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/162/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [316K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-l
[jira] [Created] (YARN-5921) Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState
Varun Saxena created YARN-5921:
--
Summary: Incorrect synchronization in
RMContextImpl#setHAServiceState/getHAServiceState
Key: YARN-5921
URL: https://issues.apache.org/jira/browse/YARN-5921
Project: Hadoop YARN
Issue Type: Bug
Reporter: Varun Saxena
Assignee: Varun Saxena
Code in RMContextImpl is as under:
{code:title=RMContextImpl.java|borderStyle=solid}
void setHAServiceState(HAServiceState haServiceState) {
synchronized (haServiceState) {
this.haServiceState = haServiceState;
}
}
public HAServiceState getHAServiceState() {
synchronized (haServiceState) {
return haServiceState;
}
}
{code}
As can be seen above, in setHAServiceState, we are synchronizing on the passed
haServiceState instead of haServiceState in RMContextImpl which will not lead
to desired effect. This does not seem to be intentional.
We can use a RW lock or synchronize on some object here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5922) Remove direct references of HBaseTimelineWriter/Reader in core ATS classes
Haibo Chen created YARN-5922: Summary: Remove direct references of HBaseTimelineWriter/Reader in core ATS classes Key: YARN-5922 URL: https://issues.apache.org/jira/browse/YARN-5922 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Affects Versions: 3.0.0-alpha1 Reporter: Haibo Chen Assignee: Haibo Chen -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5923) Unable to access logs for a running application if YARN_ACL_ENABLE is enabled
Xuan Gong created YARN-5923: --- Summary: Unable to access logs for a running application if YARN_ACL_ENABLE is enabled Key: YARN-5923 URL: https://issues.apache.org/jira/browse/YARN-5923 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
