[jira] [Commented] (YARN-4743) ResourceManager crash because TimSort
[ https://issues.apache.org/jira/browse/YARN-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531924#comment-15531924 ] Zephyr Guo commented on YARN-4743: -- Thanks for you comment.I will submit a new patch in recent days. > ResourceManager crash because TimSort > - > > Key: YARN-4743 > URL: https://issues.apache.org/jira/browse/YARN-4743 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.4 >Reporter: Zephyr Guo > Fix For: 3.0.0-alpha1 > > Attachments: YARN-4743-v1.patch, YARN-CDH5.4.7.patch, timsort.log > > > {code} > 2016-02-26 14:08:50,821 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: Comparison method violates its general > contract! > at java.util.TimSort.mergeHi(TimSort.java:868) > at java.util.TimSort.mergeAt(TimSort.java:485) > at java.util.TimSort.mergeCollapse(TimSort.java:410) > at java.util.TimSort.sort(TimSort.java:214) > at java.util.TimSort.sort(TimSort.java:173) > at java.util.Arrays.sort(Arrays.java:659) > at java.util.Collections.sort(Collections.java:217) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:316) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:240) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1091) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:989) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1185) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:684) > at java.lang.Thread.run(Thread.java:745) > 2016-02-26 14:08:50,822 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, bbye.. > {code} > Actually, this issue found in 2.6.0-cdh5.4.7. > I think the cause is that we modify {{Resouce}} while we are sorting > {{runnableApps}}. > {code:title=FSLeafQueue.java} > Comparator comparator = policy.getComparator(); > writeLock.lock(); > try { > Collections.sort(runnableApps, comparator); > } finally { > writeLock.unlock(); > } > readLock.lock(); > {code} > {code:title=FairShareComparator} > public int compare(Schedulable s1, Schedulable s2) { > .. > s1.getResourceUsage(), minShare1); > boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null, > s2.getResourceUsage(), minShare2); > minShareRatio1 = (double) s1.getResourceUsage().getMemory() > / Resources.max(RESOURCE_CALCULATOR, null, minShare1, > ONE).getMemory(); > minShareRatio2 = (double) s2.getResourceUsage().getMemory() > / Resources.max(RESOURCE_CALCULATOR, null, minShare2, > ONE).getMemory(); > .. > {code} > {{getResourceUsage}} will return current Resource. The current Resource is > unstable. > {code:title=FSAppAttempt.java} > @Override > public Resource getResourceUsage() { > // Here the getPreemptedResources() always return zero, except in > // a preemption round > return Resources.subtract(getCurrentConsumption(), > getPreemptedResources()); > } > {code} > {code:title=SchedulerApplicationAttempt} > public Resource getCurrentConsumption() { > return currentConsumption; > } > // This method may modify current Resource. > public synchronized void recoverContainer(RMContainer rmContainer) { > .. > Resources.addTo(currentConsumption, rmContainer.getContainer() > .getResource()); > .. > } > {code} > I suggest that use stable Resource in comparator. > Is there something i think wrong? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5145) [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR
[ https://issues.apache.org/jira/browse/YARN-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531918#comment-15531918 ] Sunil G commented on YARN-5145: --- I think hosting new web UI under same old webapp port can be tracked via another ticket. I ll raise one ticket if there are no objections. > [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR > - > > Key: YARN-5145 > URL: https://issues.apache.org/jira/browse/YARN-5145 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Kai Sasaki > Attachments: 0001-YARN-5145-Run-NewUI-WithOldPort-POC.patch, > YARN-5145-YARN-3368.01.patch, newUIInOldRMWebServer.png > > > Existing YARN UI configuration is under Hadoop package's directory: > $HADOOP_PREFIX/share/hadoop/yarn/webapps/, we should move it to > $HADOOP_CONF_DIR like other configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3139) Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler
[ https://issues.apache.org/jira/browse/YARN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531876#comment-15531876 ] Hadoop QA commented on YARN-3139: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 26s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 57s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 242 unchanged - 54 fixed = 248 total (was 296) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 26s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 50m 0s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830850/YARN-3139.6.patch | | JIRA Issue | YARN-3139 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 49cd0d18bfa6 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 47f8092 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13247/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/13247/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit test logs | https://builds.apache.org/job/PreCommit-YARN-Build/13247/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results
[jira] [Updated] (YARN-4327) RM can not renew TIMELINE_DELEGATION_TOKEN in secure clusters
[ https://issues.apache.org/jira/browse/YARN-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated YARN-4327: -- Component/s: security > RM can not renew TIMELINE_DELEGATION_TOKEN in secure clusters > -- > > Key: YARN-4327 > URL: https://issues.apache.org/jira/browse/YARN-4327 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, security, timelineserver >Affects Versions: 2.7.1 > Environment: hadoop 2.7.1hdfs,yarn, mrhistoryserver, ATS all use > kerberos security. > conf like this: > > hadoop.security.authorization > true > Is service-level authorization enabled? > > > hadoop.security.authentication > kerberos > Possible values are simple (no authentication), and kerberos > > >Reporter: zhangshilong > > bin hadoop 2.7.1 > ATS conf like this: > > yarn.timeline-service.http-authentication.type > simple > > > yarn.timeline-service.http-authentication.kerberos.principal > HTTP/_h...@xxx.com > > > yarn.timeline-service.http-authentication.kerberos.keytab > /etc/hadoop/keytabs/xxx.keytab > > > yarn.timeline-service.principal > xxx/_h...@xxx.com > > > yarn.timeline-service.keytab > /etc/hadoop/keytabs/xxx.keytab > > > yarn.timeline-service.best-effort > true > > > yarn.timeline-service.enabled > true > > > I'd like to allow everyone to access ATS from HTTP as RM,HDFS. > client can submit job to RM and add TIMELINE_DELEGATION_TOKEN to AM > Context, but RM can not renew TIMELINE_DELEGATION_TOKEN and make application > to failure. > RM logs: > 2015-11-03 11:58:38,191 WARN > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: > Unable to add the application to the delegation token renewer. > java.io.IOException: Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, > Service: 10.12.38.4:8188, Ident: (owner=yarn-test, renewer=yarn-test, > realUser=, issueDate=1446523118046, maxDate=1447127918046, sequenceNumber=9, > masterKeyId=2) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:439) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:78) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:847) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:828) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: HTTP status [500], message [Null user] > at > org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:169) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:287) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:212) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:414) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$3.run(TimelineClientImpl.java:396) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$3.run(TimelineClientImpl.java:378) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$5.run(TimelineClientImpl.java:451) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:183) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:466) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.renewDelegationToken(TimelineClientImpl.java:400) > at > org.apache.hadoop.yarn.security.client.TimelineDelegationTokenIdentifier$Renewer.renew(TimelineDelegationTokenIdentifier.java:81) > at org.apache.hadoop.security.token.Token.renew(Token.java:377) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationToken
[jira] [Updated] (YARN-3139) Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler
[ https://issues.apache.org/jira/browse/YARN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3139: - Attachment: YARN-3139.6.patch [~jianhe], added the null check, somehow I forgot updating this. Fixed this in ver.6 patch, mind to review again? > Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler > -- > > Key: YARN-3139 > URL: https://issues.apache.org/jira/browse/YARN-3139 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-3139.0.patch, YARN-3139.1.patch, YARN-3139.2.patch, > YARN-3139.3.patch, YARN-3139.4.patch, YARN-3139.5.patch, YARN-3139.6.patch > > > Enhance locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler, as > mentioned in YARN-3091, a possible solution is using read/write lock. Other > fine-graind locks for specific purposes / bugs should be addressed in > separated tickets. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531569#comment-15531569 ] Tao Yang commented on YARN-5683: Thanks for comments, [~grey]. We think [YARN-3409|https://issues.apache.org/jira/browse/YARN-3409] is helpful for applications to be allocated on the required nodes and will meet the demands of some other requirements except this. We also pay attention to [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139]. SSD is tight on heterogeneous clusters and might be over-allocated without constraints. Take it as a resource may be a good solution and we will discuss the support of storage type in this feature later. > Support specifying storage type for per-application local dirs > -- > > Key: YARN-5683 > URL: https://issues.apache.org/jira/browse/YARN-5683 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Tao Yang > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5683-1.patch, flow_diagram_for_MapReduce.png > > > h3. Introduction > * Some applications of various frameworks (Flink, Spark and MapReduce etc) > using local storage (checkpoint, shuffle etc) might require high IO > performance. It's useful to allocate local directories to high performance > storage media for these applications on heterogeneous clusters. > * YARN does not distinguish different storage types and hence applications > cannot selectively use storage media with different performance > characteristics. Adding awareness of storage media can allow YARN to make > better decisions about the placement of local directories. > h3. Approach > * NodeManager will distinguish storage types for local directories. > ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration > should allow the cluster administrator to optionally specify the storage type > for each local directories. Example: > [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to > [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) > ** StorageType defines DISK/SSD storage types and takes DISK as the default > storage type. > ** StorageLocation separates storage type and directory path, used by > LocalDirAllocator to aware the types of local dirs, the default storage type > is DISK. > ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the > local directory of the specified storage type, and will fallback to not care > storage type if the requirement can not be satisfied. > ** Support for container related local/log directories by ContainerLaunch. > All application frameworks can set the environment variables > (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage > type of local/log directories. > * Allow specified storage type for various frameworks (Take MapReduce as an > example) > ** Add new configurations should allow application administrator to > optionally specify the storage type of local/log directories. (MapReduce add > configurations: mapreduce.job.local-storage-type and > mapreduce.job.log-storage-type) > ** Support for container work directories. Set the environment variables > includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations > above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce > should update YARNRunner and TaskAttemptImpl) > ** Add storage type prefix for request path to support for other local > directories of frameworks (such as shuffle directories for MapReduce). > (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to > support for output/work directories) > ** Flow diagram for MapReduce framework > !flow_diagram_for_MapReduce.png! > h3. Further Discussion > * The requirement of storage type for local/log directories may not be > satisfied on heterogeneous clusters. To achieve global optimum, scheduler > should aware and manage disk resources. > [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that > but seems not support multiple storage types, maybe we should do even more to > aware the storage type of disk resource? > * Node labels or node constraints > ([YARN-3409|https://issues.apache.org/jira/browse/YARN-3409]) can also make a > higher chance to satisfy the requirement of specified storage type. > * Fallback strategy still needs to be concerned. Certain applications might > not work well when the requirement of storage type is not satisfied. When > none of desired storage type disk are available, should container launching > be failed? let AM handle? -- This message was sent by Atlassian JIRA (v6.3.4#6332) -
[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-5683: --- Description: h3. Introduction * Some applications of various frameworks (Flink, Spark and MapReduce etc) using local storage (checkpoint, shuffle etc) might require high IO performance. It's useful to allocate local directories to high performance storage media for these applications on heterogeneous clusters. * YARN does not distinguish different storage types and hence applications cannot selectively use storage media with different performance characteristics. Adding awareness of storage media can allow YARN to make better decisions about the placement of local directories. h3. Approach * NodeManager will distinguish storage types for local directories. ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration should allow the cluster administrator to optionally specify the storage type for each local directories. Example: [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) ** StorageType defines DISK/SSD storage types and takes DISK as the default storage type. ** StorageLocation separates storage type and directory path, used by LocalDirAllocator to aware the types of local dirs, the default storage type is DISK. ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the local directory of the specified storage type, and will fallback to not care storage type if the requirement can not be satisfied. ** Support for container related local/log directories by ContainerLaunch. All application frameworks can set the environment variables (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage type of local/log directories. * Allow specified storage type for various frameworks (Take MapReduce as an example) ** Add new configurations should allow application administrator to optionally specify the storage type of local/log directories. (MapReduce add configurations: mapreduce.job.local-storage-type and mapreduce.job.log-storage-type) ** Support for container work directories. Set the environment variables includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce should update YARNRunner and TaskAttemptImpl) ** Add storage type prefix for request path to support for other local directories of frameworks (such as shuffle directories for MapReduce). (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to support for output/work directories) ** Flow diagram for MapReduce framework !flow_diagram_for_MapReduce.png! h3. Further Discussion * The requirement of storage type for local/log directories may not be satisfied on heterogeneous clusters. To achieve global optimum, scheduler should aware and manage disk resources. [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that but seems not support multiple storage types, maybe we should do even more to aware the storage type of disk resource? * Node labels or node constraints ([YARN-3409|https://issues.apache.org/jira/browse/YARN-3409]) can also make a higher chance to satisfy the requirement of specified storage type. * Fallback strategy still needs to be concerned. Certain applications might not work well when the requirement of storage type is not satisfied. When none of desired storage type disk are available, should container launching be failed? let AM handle? was: h3. Introduction * Some applications of various frameworks (Flink, Spark and MapReduce etc) using local storage (checkpoint, shuffle etc) might require high IO performance. It's useful to allocate local directories to high performance storage media for these applications on heterogeneous clusters. * YARN does not distinguish different storage types and hence applications cannot selectively use storage media with different performance characteristics. Adding awareness of storage media can allow YARN to make better decisions about the placement of local directories. h3. Approach * NodeManager will distinguish storage types for local directories. ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration should allow the cluster administrator to optionally specify the storage type for each local directories. Example: [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) ** StorageType defines DISK/SSD storage types and takes DISK as the default storage type. ** StorageLocation separates storage type and directory path, used by LocalDirAllocator to aware the types of local dirs, the default storage type is DISK. ** getLocalPathForWrite method of LocalDirAl
[jira] [Commented] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531542#comment-15531542 ] Hadoop QA commented on YARN-5384: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 9s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 4s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 59s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 50s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 53s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s {color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 18s {color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 50s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 6s {color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 67m 34s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.reservation.planning.TestGreedyReservationAgent | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830828/YARN-5384.v6.patch | | JIRA Issue | YARN-5384 | | Optional Tests | a
[jira] [Commented] (YARN-5486) Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests
[ https://issues.apache.org/jira/browse/YARN-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531533#comment-15531533 ] Hadoop QA commented on YARN-5486: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 7s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 43s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 41s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 6s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 18s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 36s {color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 141 unchanged - 0 fixed = 142 total (was 141) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s {color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 11s {color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 34m 46s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 9s {color} | {color:green} hadoop-yarn-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 100m 21s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830811/YARN-5486.004.patch | | JIRA Issue | YARN-5486 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 76bdc93c098c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 47f8092 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13243/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
[jira] [Commented] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531522#comment-15531522 ] Hadoop QA commented on YARN-5384: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 1s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 7s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 2s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 54s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 54s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 54s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s {color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 31 unchanged - 0 fixed = 32 total (was 31) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 54s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 8s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 32s {color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 43s {color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 30s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 8s {color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 81m 45s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | | | hadoop.yarn.server.resourcemanager.reservation.planning.TestGreedyReservationAgent | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hado
[jira] [Commented] (YARN-5677) RM can be in active-active state for an extended period
[ https://issues.apache.org/jira/browse/YARN-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531486#comment-15531486 ] Hadoop QA commented on YARN-5677: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 56s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 3s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 50s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 48m 41s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830814/YARN-5677.001.patch | | JIRA Issue | YARN-5677 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 378e0f79203f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 47f8092 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13245/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/13245/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > RM can be in active-active state for an extended period > --- > > Key: YARN-5677 > URL: https://issues.apache.org/jira/browse/YARN-5677 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.0.0-alpha1 >Report
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v6.patch YARN-5384.v6.patch fixes the checkstyle issues. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, > YARN-5384.v3.patch, YARN-5384.v4.patch, YARN-5384.v5.patch, YARN-5384.v6.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531411#comment-15531411 ] Hadoop QA commented on YARN-5384: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 45s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 5s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 15s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s {color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 132 unchanged - 0 fixed = 133 total (was 132) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 7s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 58s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 3s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s {color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 29s {color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 35m 1s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 0s {color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 7s {color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 89m 28s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.reservation.planning.TestGreedyReservationAgent |
[jira] [Updated] (YARN-5677) RM can be in active-active state for an extended period
[ https://issues.apache.org/jira/browse/YARN-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5677: --- Attachment: YARN-5677.001.patch This patch fixes the issue in trunk. I opted to be conservative and wait out the ZK session timeout rather than failing over immediately. The delay extends the period of time that the cluster is in active-active, but it hopefully reduces jitter in the face of minor network disturbances. I'll need to post an entirely different fix for branch-2.7. Should I open a second JIRA for that? > RM can be in active-active state for an extended period > --- > > Key: YARN-5677 > URL: https://issues.apache.org/jira/browse/YARN-5677 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.3, 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5677.001.patch > > > Both branch-2.8/trunk and branch-2.7 have issues when the active RM loses > contact with the ZK node(s). > In branch-2.7, the RM will retry the connection 1000 times by default. > Attempting to contact a node which cannot be reached is slow, which means the > active can take over an hour to realize it is no longer active. I clocked it > at about an hour and a half in my tests. The solution appears to be to add > some time awareness into the retry loop. > In branch-2.8/trunk, there is no maximum number of retries that I see. It > appears the connection will be retried forever, with the active never > figuring out it's no longer active. In my testing, the active-active state > lasted almost 2 hours with no sign of stopping before I killed it. The > solution appears to be to cap the number of retries or amount of time spent > retrying. > This issue is significant because of the asynchronous nature of job > submission. If the active doesn't know it's not active, it will buffer up > job submissions until it finally realizes it has become the standby. Then it > will fail all the job submissions in bulk. In high-volume workflows, that > behavior can create huge mass job failures. > This issue is also important because the node managers will not fail over to > the new active until the old active realizes it's the standby. Workloads > submitted after the old active loses contact with ZK will therefore fail to > be executed regardless of which RM the clients contact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5688) Make allocation of opportunistic containers asynchronous
Konstantinos Karanasos created YARN-5688: Summary: Make allocation of opportunistic containers asynchronous Key: YARN-5688 URL: https://issues.apache.org/jira/browse/YARN-5688 Project: Hadoop YARN Issue Type: Sub-task Reporter: Konstantinos Karanasos In the current implementation of the {{OpportunisticContainerAllocatorAMService}}, we synchronously perform the allocation of opportunistic containers. This results in "blocking" the service at the RM when scheduling the opportunistic containers. The {{OpportunisticContainerAllocator}} should instead asynchronously run as a separate thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5687) Refactor TestOpportunisticContainerAllocation to extend TestAMRMClient
Konstantinos Karanasos created YARN-5687: Summary: Refactor TestOpportunisticContainerAllocation to extend TestAMRMClient Key: YARN-5687 URL: https://issues.apache.org/jira/browse/YARN-5687 Project: Hadoop YARN Issue Type: Sub-task Reporter: Konstantinos Karanasos Since {{TestOpportunisticContainerAllocation}} shares a lot of code with the {{TestAMRMClient}}, we should refactor the former, making it a subclass of the latter. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5486) Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests
[ https://issues.apache.org/jira/browse/YARN-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-5486: - Attachment: YARN-5486.004.patch Adding new patch. Fixed remaining checkstyle issues and [~asuresh]'s comments. The testcase is not failing locally for me and does not seem related. [~asuresh], I will create JIRAs to track the two issues you mentioned. Good point about the opportunistic container allocation. We should make it asynchronous. > Update OpportunisticContainerAllocatorAMService::allocate method to handle > OPPORTUNISTIC container requests > --- > > Key: YARN-5486 > URL: https://issues.apache.org/jira/browse/YARN-5486 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos > Attachments: YARN-5486.001.patch, YARN-5486.002.patch, > YARN-5486.003.patch, YARN-5486.004.patch > > > YARN-5457 refactors the Distributed Scheduling framework to move the > container allocator to yarn-server-common. > This JIRA proposes to update the allocate method in the new AM service to use > the OpportunisticContainerAllocator to allocate opportunistic containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v5.patch Thanks [~subru] for your comments! YARN-5384.v5.patch addresses all your comments, and removes some unrelated changes from YARN-5384.v4.patch, > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, > YARN-5384.v3.patch, YARN-5384.v4.patch, YARN-5384.v5.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5686) DefaultContainerExecutor random working dir algorigthm skews results
Miklos Szegedi created YARN-5686: Summary: DefaultContainerExecutor random working dir algorigthm skews results Key: YARN-5686 URL: https://issues.apache.org/jira/browse/YARN-5686 Project: Hadoop YARN Issue Type: Bug Reporter: Miklos Szegedi Priority: Minor {code} long randomPosition = RandomUtils.nextLong() % totalAvailable; ... while (randomPosition > availableOnDisk[dir]) { randomPosition -= availableOnDisk[dir++]; } {code} The code above selects a disk based on the random number weighted by the free space on each disk respectively. For example, if I have two disks with 100 bytes each, totalAvailable is 200. The value of randomPosition will be 0..199. 0..99 should select the first disk, 100..199 should select the second disk inclusively. Random number 100 should select the second disk to be fair but this is not the case right now. We need to use {code} while (randomPosition >= availableOnDisk[dir]) {code} instead of {code} while (randomPosition > availableOnDisk[dir]) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531223#comment-15531223 ] Arpit Agarwal commented on YARN-5400: - No problem and thank you for the quick fix Robert. I verified that your latest commit fixed branch-2. > Light cleanup in ZKRMStateStore > --- > > Key: YARN-5400 > URL: https://issues.apache.org/jira/browse/YARN-5400 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5400.001.patch, YARN-5400.002.patch, > YARN-5400.003.patch, YARN-5400.addendum.patch > > > of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some > icky bits, like unused variables. This JIRA is to clean that up. It should > have no functional impact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2995) Enhance UI to show cluster resource utilization of various container types
[ https://issues.apache.org/jira/browse/YARN-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantinos Karanasos updated YARN-2995: - Attachment: YARN-2995.001.patch Adding first version of the patch. > Enhance UI to show cluster resource utilization of various container types > -- > > Key: YARN-2995 > URL: https://issues.apache.org/jira/browse/YARN-2995 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sriram Rao >Assignee: Konstantinos Karanasos > Attachments: YARN-2995.001.patch > > > This JIRA proposes to extend the Resource manager UI to show how cluster > resources are being used to run *guaranteed start* and *queueable* > containers. For example, a graph that shows over time, the fraction of > running containers that are *guaranteed start* and the fraction of running > containers that are *queueable*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531187#comment-15531187 ] Hudson commented on YARN-5400: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10509 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10509/]) YARN-5400. Light cleanup in ZKRMStateStore (templedf via rkanter) (rkanter: rev bcb2528a51c33e4caff8d744c5e14c1accfc47d0) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMZKUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java > Light cleanup in ZKRMStateStore > --- > > Key: YARN-5400 > URL: https://issues.apache.org/jira/browse/YARN-5400 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5400.001.patch, YARN-5400.002.patch, > YARN-5400.003.patch, YARN-5400.addendum.patch > > > of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some > icky bits, like unused variables. This JIRA is to clean that up. It should > have no functional impact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v4.patch > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, > YARN-5384.v3.patch, YARN-5384.v4.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5400) Light cleanup in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-5400: Attachment: YARN-5400.addendum.patch Ok, it should be fixed now. Sorry about that. I've attached the addendum patch for reference. > Light cleanup in ZKRMStateStore > --- > > Key: YARN-5400 > URL: https://issues.apache.org/jira/browse/YARN-5400 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5400.001.patch, YARN-5400.002.patch, > YARN-5400.003.patch, YARN-5400.addendum.patch > > > of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some > icky bits, like unused variables. This JIRA is to clean that up. It should > have no functional impact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5400) Light cleanup in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531129#comment-15531129 ] Robert Kanter edited comment on YARN-5400 at 9/28/16 10:57 PM: --- Yup. I was just talking to [~templedf] about that. It's a problem with the JDK 7 compiler not being able to infer the diamond operator types as well as the JDK 8 compiler does. I'll push an addendum patch with the explicit types shortly. was (Author: rkanter): Yup. I was just talking to [~templedf] about that. It's a problem with the JDK 7 compiler not being able to infer the diamond operator types as well as the JDK 8 compiler. I'll push an addendum patch with the explicit types shortly. > Light cleanup in ZKRMStateStore > --- > > Key: YARN-5400 > URL: https://issues.apache.org/jira/browse/YARN-5400 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5400.001.patch, YARN-5400.002.patch, > YARN-5400.003.patch > > > of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some > icky bits, like unused variables. This JIRA is to clean that up. It should > have no functional impact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531129#comment-15531129 ] Robert Kanter commented on YARN-5400: - Yup. I was just talking to [~templedf] about that. It's a problem with the JDK 7 compiler not being able to infer the diamond operator types as well as the JDK 8 compiler. I'll push an addendum patch with the explicit types shortly. > Light cleanup in ZKRMStateStore > --- > > Key: YARN-5400 > URL: https://issues.apache.org/jira/browse/YARN-5400 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5400.001.patch, YARN-5400.002.patch, > YARN-5400.003.patch > > > of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some > icky bits, like unused variables. This JIRA is to clean that up. It should > have no functional impact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5466) DefaultContainerExecutor needs JavaDocs
[ https://issues.apache.org/jira/browse/YARN-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531132#comment-15531132 ] Miklos Szegedi commented on YARN-5466: -- looks good to me +1(non-binding) > DefaultContainerExecutor needs JavaDocs > --- > > Key: YARN-5466 > URL: https://issues.apache.org/jira/browse/YARN-5466 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: YARN-5466.001.patch > > > Following on YARN-5455, let's document the DefaultContainerExecutor as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531122#comment-15531122 ] Arpit Agarwal commented on YARN-5400: - Looks like this commit broke the build for branch-2. [~rkanter] can you please take a look? {code} [ERROR] COMPILATION ERROR : [ERROR] /Users/aagarwal/src/commit/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java:[415,40] method put in interface java.util.Map cannot be applied to given types; required: java.lang.String,java.util.Map found: java.lang.String,java.util.HashMap reason: actual argument java.util.HashMap cannot be converted to java.util.Map by method invocation conversion {code} > Light cleanup in ZKRMStateStore > --- > > Key: YARN-5400 > URL: https://issues.apache.org/jira/browse/YARN-5400 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5400.001.patch, YARN-5400.002.patch, > YARN-5400.003.patch > > > of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some > icky bits, like unused variables. This JIRA is to clean that up. It should > have no functional impact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5685) Non-embedded HA failover is broken
[ https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530963#comment-15530963 ] Daniel Templeton commented on YARN-5685: BTW, the bug introduced by YARN-4559 (that I had in the original description) would break non-embedded failover if it worked, but since it doesn't, it's kinda harmless. Still wrong, though. > Non-embedded HA failover is broken > -- > > Key: YARN-5685 > URL: https://issues.apache.org/jira/browse/YARN-5685 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0, 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > > If HA is enabled with automatic failover enabled and embedded failover > disabled, all RMs all come up in standby state. To make one of them active, > the {{--forcemanual}} flag must be used when manually triggering the state > change. Should the active go down, the standby will not become active and > must be manually transitioned with the {{--forcemanual}} flag. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5685) Non-embedded HA failover is broken
[ https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5685: --- Description: If HA is enabled with automatic failover enabled and embedded failover disabled, all RMs all come up in standby state. To make one of them active, the {{--forcemanual}} flag must be used when manually triggering the state change. Should the active go down, the standby will not become active and must be manually transitioned with the {{--forcemanual}} flag. (was: YARN-4559 broke RM HA when embedded automatic failover is disabled. The {{ZKRMStateStore}} will now only start its monitoring thread when automatic failover not enabled (which is patently useless). I presume the intended change was to have the monitoring thread started when automatic failover is not *embedded*. If HA is enabled with automatic failover enabled and embedded failover disabled, all RMs all come up in standby state. To make one of them active, the {{--forcemanual}} flag must be used when manually triggering the state change. Should the active go down, the standby will not become active and must be manually transitioned with the {{--forcemanual}} flag.) > Non-embedded HA failover is broken > -- > > Key: YARN-5685 > URL: https://issues.apache.org/jira/browse/YARN-5685 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0, 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > > If HA is enabled with automatic failover enabled and embedded failover > disabled, all RMs all come up in standby state. To make one of them active, > the {{--forcemanual}} flag must be used when manually triggering the state > change. Should the active go down, the standby will not become active and > must be manually transitioned with the {{--forcemanual}} flag. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5685) Non-embedded HA failover is broken
[ https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530951#comment-15530951 ] Daniel Templeton commented on YARN-5685: I think I finally get it. The net is that HA now requires either Curator or embedded to be enabled. Letting the {{ZKRMStateStore}} handle the RM state transitions is no longer an option and hasn't been for a while. There are only three ways {{transitionToActive()}} gets called: # {{EmbeddedElectorService.becomeActive()}} -- embedded failover # {{LeaderElectorService.isLeader()}} -- curator failover # {{AdminService.transitionToActive()}} -- CLI The question is what to do about non-embedded failover. I don't think it's worth making non-embedded failover work again. I think we're better off declaring non-embedded failover to be dead and removing the option altogether. Given that this has been broken for a while and no one has noticed, I'm going to bump the priority down a bit. > Non-embedded HA failover is broken > -- > > Key: YARN-5685 > URL: https://issues.apache.org/jira/browse/YARN-5685 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0, 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > > YARN-4559 broke RM HA when embedded automatic failover is disabled. The > {{ZKRMStateStore}} will now only start its monitoring thread when automatic > failover not enabled (which is patently useless). I presume the intended > change was to have the monitoring thread started when automatic failover is > not *embedded*. > If HA is enabled with automatic failover enabled and embedded failover > disabled, all RMs all come up in standby state. To make one of them active, > the {{--forcemanual}} flag must be used when manually triggering the state > change. Should the active go down, the standby will not become active and > must be manually transitioned with the {{--forcemanual}} flag. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5685) Non-embedded HA failover is broken
[ https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530951#comment-15530951 ] Daniel Templeton edited comment on YARN-5685 at 9/28/16 9:37 PM: - I think I finally get it. The net is that HA now requires either Curator or embedded to be enabled. Letting the {{ZKRMStateStore}} handle the RM state transitions is no longer an option and hasn't been for a while. There are only three ways {{transitionToActive()}} gets called: # {{EmbeddedElectorService.becomeActive()}} -- embedded failover # {{LeaderElectorService.isLeader()}} -- curator failover # {{AdminService.transitionToActive()}} -- CLI The question is what to do about non-embedded failover. I don't think it's worth making non-embedded failover work again. I think we're better off declaring non-embedded failover to be dead and removing the option altogether. was (Author: templedf): I think I finally get it. The net is that HA now requires either Curator or embedded to be enabled. Letting the {{ZKRMStateStore}} handle the RM state transitions is no longer an option and hasn't been for a while. There are only three ways {{transitionToActive()}} gets called: # {{EmbeddedElectorService.becomeActive()}} -- embedded failover # {{LeaderElectorService.isLeader()}} -- curator failover # {{AdminService.transitionToActive()}} -- CLI The question is what to do about non-embedded failover. I don't think it's worth making non-embedded failover work again. I think we're better off declaring non-embedded failover to be dead and removing the option altogether. Given that this has been broken for a while and no one has noticed, I'm going to bump the priority down a bit. > Non-embedded HA failover is broken > -- > > Key: YARN-5685 > URL: https://issues.apache.org/jira/browse/YARN-5685 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0, 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > > YARN-4559 broke RM HA when embedded automatic failover is disabled. The > {{ZKRMStateStore}} will now only start its monitoring thread when automatic > failover not enabled (which is patently useless). I presume the intended > change was to have the monitoring thread started when automatic failover is > not *embedded*. > If HA is enabled with automatic failover enabled and embedded failover > disabled, all RMs all come up in standby state. To make one of them active, > the {{--forcemanual}} flag must be used when manually triggering the state > change. Should the active go down, the standby will not become active and > must be manually transitioned with the {{--forcemanual}} flag. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530923#comment-15530923 ] Robert Kanter commented on YARN-5400: - +1 > Light cleanup in ZKRMStateStore > --- > > Key: YARN-5400 > URL: https://issues.apache.org/jira/browse/YARN-5400 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Attachments: YARN-5400.001.patch, YARN-5400.002.patch, > YARN-5400.003.patch > > > of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some > icky bits, like unused variables. This JIRA is to clean that up. It should > have no functional impact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530861#comment-15530861 ] Hadoop QA commented on YARN-5400: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 4s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s {color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 52 unchanged - 10 fixed = 52 total (was 62) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s {color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 935 unchanged - 6 fixed = 935 total (was 941) {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m 42s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 53m 23s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830776/YARN-5400.003.patch | | JIRA Issue | YARN-5400 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 2fc163e7875a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / c3b235e | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13241/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/13241/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Light cleanup in ZKRMStateStore > --- > > Key: YARN-5400 >
[jira] [Updated] (YARN-5678) Misleading log message in FSLeafQueue and FSParentQueue
[ https://issues.apache.org/jira/browse/YARN-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-5678: --- Summary: Misleading log message in FSLeafQueue and FSParentQueue (was: Misleading log message in FSLeafQueue) > Misleading log message in FSLeafQueue and FSParentQueue > --- > > Key: YARN-5678 > URL: https://issues.apache.org/jira/browse/YARN-5678 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.0.0-alpha1 >Reporter: Yufei Gu >Assignee: Yufei Gu > > {code} > private void updateDemandForApp(FSAppAttempt sched, Resource maxRes) { > sched.updateDemand(); > Resource toAdd = sched.getDemand(); > if (LOG.isDebugEnabled()) { > LOG.debug("Counting resource from " + sched.getName() + " " + toAdd > + "; Total resource consumption for " + getName() + " now " > + demand); > } > demand = Resources.add(demand, toAdd); > demand = Resources.componentwiseMin(demand, maxRes); > } > {code} > Change the "resource consumption" to "resource demand". -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5400) Light cleanup in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5400: --- Attachment: YARN-5400.003.patch Here's a rebased version. > Light cleanup in ZKRMStateStore > --- > > Key: YARN-5400 > URL: https://issues.apache.org/jira/browse/YARN-5400 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Attachments: YARN-5400.001.patch, YARN-5400.002.patch, > YARN-5400.003.patch > > > of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some > icky bits, like unused variables. This JIRA is to clean that up. It should > have no functional impact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530688#comment-15530688 ] Robert Kanter commented on YARN-5400: - Hmm, it doesn't apply cleanly now: {noformat} >> [27] 12:51 : hadoop-git (trunk) :: patch -p0 < >> ~/Downloads/YARN-5400.002.patch patching file hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMZKUtils.java patching file hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java patching file hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java Hunk #34 FAILED at 764. Hunk #35 succeeded at 801 (offset -3 lines). Hunk #36 succeeded at 827 (offset -3 lines). Hunk #37 succeeded at 860 (offset -3 lines). Hunk #38 succeeded at 881 (offset -3 lines). Hunk #39 succeeded at 910 (offset -3 lines). Hunk #40 succeeded at 987 (offset -3 lines). Hunk #41 succeeded at 1024 (offset -3 lines). 1 out of 41 hunks FAILED -- saving rejects to file hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java.rej {noformat} > Light cleanup in ZKRMStateStore > --- > > Key: YARN-5400 > URL: https://issues.apache.org/jira/browse/YARN-5400 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Attachments: YARN-5400.001.patch, YARN-5400.002.patch > > > of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some > icky bits, like unused variables. This JIRA is to clean that up. It should > have no functional impact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore
[ https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530671#comment-15530671 ] Robert Kanter commented on YARN-5400: - Will do momentarily. > Light cleanup in ZKRMStateStore > --- > > Key: YARN-5400 > URL: https://issues.apache.org/jira/browse/YARN-5400 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Trivial > Attachments: YARN-5400.001.patch, YARN-5400.002.patch > > > of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some > icky bits, like unused variables. This JIRA is to clean that up. It should > have no functional impact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5599) Publish AM launch command to ATS
[ https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530666#comment-15530666 ] Hadoop QA commented on YARN-5599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 36s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 30s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 19s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 57s {color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 16s {color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 3s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 57s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 16s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s {color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s {color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 13s {color} | {color:green} the patch passed with JDK v1.7.0_111 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 13s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s {color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 232 unchanged - 1 fixed = 233 total (was 233) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 58s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s {color} | {color:green} the patch passed with JDK v1.7.0_111 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s {color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_101. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s {color} | {color:green} hadoop-yarn-common in the patch passed with JDK v1.8.0_101. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s {color} | {color:green} hadoop-yarn-server-common in the patch passed with JDK v1.8.0_101. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 9s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed with JDK v1.8.0_101. {color} | | {c
[jira] [Commented] (YARN-5685) Non-embedded HA failover is broken
[ https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530642#comment-15530642 ] Daniel Templeton commented on YARN-5685: The issue is more than that change from YARN-4559. I'm still digging, but even with that issue resolved the RMs are still all stuck in standby because the state store isn't started until the RM transitions to active, but it doesn't transition to active unless the state store is started. > Non-embedded HA failover is broken > -- > > Key: YARN-5685 > URL: https://issues.apache.org/jira/browse/YARN-5685 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0, 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > > YARN-4559 broke RM HA when embedded automatic failover is disabled. The > {{ZKRMStateStore}} will now only start its monitoring thread when automatic > failover not enabled (which is patently useless). I presume the intended > change was to have the monitoring thread started when automatic failover is > not *embedded*. > If HA is enabled with automatic failover enabled and embedded failover > disabled, all RMs all come up in standby state. To make one of them active, > the {{--forcemanual}} flag must be used when manually triggering the state > change. Should the active go down, the standby will not become active and > must be manually transitioned with the {{--forcemanual}} flag. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4329) Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530633#comment-15530633 ] Hadoop QA commented on YARN-4329: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 10s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 10s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s {color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 938 unchanged - 3 fixed = 938 total (was 941) {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 35m 18s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 53m 36s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830760/YARN-4329.002.patch | | JIRA Issue | YARN-4329 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 39b5cdbe6643 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / e19b37e | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13239/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/13239/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Allow fetching exact reason as to why a submitted app is in ACCEPTED state in > Fair Scheduler > > > Key: YARN-4329 > URL: https://issues.apache.org/jira/browse/YARN-4329 > Project: Hadoop YARN > Issue Type: Sub-task >
[jira] [Commented] (YARN-4329) Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530632#comment-15530632 ] Daniel Templeton commented on YARN-4329: Thanks, [~yufei]. Maybe I'm an old codger, but I'd really rather you have a single return point from {{FSAppAttempt.hasContainerForNode()}} and {{MaxRunningAppsEnforcer.canAppBeRunnable()}}. There's no advantage to having multiple exit points in this case, and it makes maintenance a tiny bit harder. > Allow fetching exact reason as to why a submitted app is in ACCEPTED state in > Fair Scheduler > > > Key: YARN-4329 > URL: https://issues.apache.org/jira/browse/YARN-4329 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler, resourcemanager >Reporter: Naganarasimha G R >Assignee: Yufei Gu > Attachments: YARN-4329.001.patch, YARN-4329.002.patch > > > Similar to YARN-3946, it would be useful to capture possible reason why the > Application is in accepted state in FairScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4561) Compaction coprocessor enhancements: On/Off, whitelisting, blacklisting
[ https://issues.apache.org/jira/browse/YARN-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530597#comment-15530597 ] Joep Rottinghuis commented on YARN-4561: All data that comes in late would be marked as suspicious. Only if accompanied with non-suspicious data will we keep this. If there is no other normal of final-value record we can discard this suspicious data. > Compaction coprocessor enhancements: On/Off, whitelisting, blacklisting > --- > > Key: YARN-4561 > URL: https://issues.apache.org/jira/browse/YARN-4561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: YARN-5355 > > YARN-4062 deals with the flush and compaction related coprocessor basic > functionality. We also need to ensure we can turn compaction on/off as a > whole (in case of dealing with production issues) as well as provide a way to > allow for blacklisting and whitelisting of processing compaction for certain > records. > For instance, we may want to compact only those records which belong to > applications in that datacenter. This way we donot interfere with hbase > replication causing coprocessors to process the same record in more than one > dc at the same time. > Also, we might want to not compact/process certain records, perhaps whose > rowkey matches a certain criteria. > Filing jira to track these enhancements -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5145) [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR
[ https://issues.apache.org/jira/browse/YARN-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-5145: -- Attachment: newUIInOldRMWebServer.png > [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR > - > > Key: YARN-5145 > URL: https://issues.apache.org/jira/browse/YARN-5145 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Kai Sasaki > Attachments: 0001-YARN-5145-Run-NewUI-WithOldPort-POC.patch, > YARN-5145-YARN-3368.01.patch, newUIInOldRMWebServer.png > > > Existing YARN UI configuration is under Hadoop package's directory: > $HADOOP_PREFIX/share/hadoop/yarn/webapps/, we should move it to > $HADOOP_CONF_DIR like other configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5145) [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR
[ https://issues.apache.org/jira/browse/YARN-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-5145: -- Attachment: 0001-YARN-5145-Run-NewUI-WithOldPort-POC.patch Attaching a POC patch to host new web UI under same old webapp port. {noformat} http:///newUI/ {noformat} Also attaching a screen shot.. I think we can choose this model.. Thoughts? > [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR > - > > Key: YARN-5145 > URL: https://issues.apache.org/jira/browse/YARN-5145 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Kai Sasaki > Attachments: 0001-YARN-5145-Run-NewUI-WithOldPort-POC.patch, > YARN-5145-YARN-3368.01.patch > > > Existing YARN UI configuration is under Hadoop package's directory: > $HADOOP_PREFIX/share/hadoop/yarn/webapps/, we should move it to > $HADOOP_CONF_DIR like other configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4561) Compaction coprocessor enhancements: On/Off, whitelisting, blacklisting
[ https://issues.apache.org/jira/browse/YARN-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530564#comment-15530564 ] Joep Rottinghuis commented on YARN-4561: For late arriving data that was spooled using whatever we come up with in YARN-4061, we can still recognize records that came in after we got rid of the finalized application's last write. If we can have the _first_ write marked as a special value, just like the final value is marked, then we can later on distinguish the case where we have a later-arriving update. Aside from the very first record (marked special) we will always have an existing record that will slide in front of or behind exising values. In other words, we see either a new value and it is the first, or we see multiple values. We could use that to see if late values come in. The trick will be to see if we can do this after the fact ( in the read or flush compaction) rather than during writes. This may have to be a write-time copro that processes data only if it is later than the normal timewindow (more than a day old). For those cases we might have to do reads during writes, or at least mark records as suspicious for later analysis. > Compaction coprocessor enhancements: On/Off, whitelisting, blacklisting > --- > > Key: YARN-4561 > URL: https://issues.apache.org/jira/browse/YARN-4561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: YARN-5355 > > YARN-4062 deals with the flush and compaction related coprocessor basic > functionality. We also need to ensure we can turn compaction on/off as a > whole (in case of dealing with production issues) as well as provide a way to > allow for blacklisting and whitelisting of processing compaction for certain > records. > For instance, we may want to compact only those records which belong to > applications in that datacenter. This way we donot interfere with hbase > replication causing coprocessors to process the same record in more than one > dc at the same time. > Also, we might want to not compact/process certain records, perhaps whose > rowkey matches a certain criteria. > Filing jira to track these enhancements -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-4329) Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530473#comment-15530473 ] Yufei Gu edited comment on YARN-4329 at 9/28/16 6:33 PM: - Hi [~templedf], thanks for the review. I uploaded the new patch for your comments. I didn't change {code} diagnosticMessageBldr.append(" exceeds current queue or its parents" + " maximum resource allowed)."); {code} to two appends since concatenation of string constants is done at compile time. was (Author: yufeigu): Hi [~templedf], thanks for the review. I uploaded the new patch for your comments. I didn't change {code} diagnosticMessageBldr.append(" exceeds current queue or its parents" + " maximum resource allowed)."); {code} to two append clause since concatenation of string constants is done at compile time. > Allow fetching exact reason as to why a submitted app is in ACCEPTED state in > Fair Scheduler > > > Key: YARN-4329 > URL: https://issues.apache.org/jira/browse/YARN-4329 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler, resourcemanager >Reporter: Naganarasimha G R >Assignee: Yufei Gu > Attachments: YARN-4329.001.patch, YARN-4329.002.patch > > > Similar to YARN-3946, it would be useful to capture possible reason why the > Application is in accepted state in FairScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4329) Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-4329: --- Attachment: YARN-4329.002.patch Hi [~templedf], thanks for the review. I uploaded the new patch for your comments. I didn't change {code} diagnosticMessageBldr.append(" exceeds current queue or its parents" + " maximum resource allowed)."); {code} to two append clause since concatenation of string constants is done at compile time. > Allow fetching exact reason as to why a submitted app is in ACCEPTED state in > Fair Scheduler > > > Key: YARN-4329 > URL: https://issues.apache.org/jira/browse/YARN-4329 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler, resourcemanager >Reporter: Naganarasimha G R >Assignee: Yufei Gu > Attachments: YARN-4329.001.patch, YARN-4329.002.patch > > > Similar to YARN-3946, it would be useful to capture possible reason why the > Application is in accepted state in FairScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5659) getPathFromYarnURL should use standard methods
[ https://issues.apache.org/jira/browse/YARN-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530407#comment-15530407 ] Hitesh Shah commented on YARN-5659: --- [~sershe] Just annotate the functions that are only required by unit tests with @Private and @VisibleForTesting > getPathFromYarnURL should use standard methods > -- > > Key: YARN-5659 > URL: https://issues.apache.org/jira/browse/YARN-5659 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: YARN-5659.01.patch, YARN-5659.02.patch, > YARN-5659.03.patch, YARN-5659.04.patch, YARN-5659.04.patch, YARN-5659.patch > > > getPathFromYarnURL does some string shenanigans where standard ctors should > suffice. > There are also bugs in it e.g. passing an empty scheme to the URI ctor is > invalid, null should be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5659) getPathFromYarnURL should use standard methods
[ https://issues.apache.org/jira/browse/YARN-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530407#comment-15530407 ] Hitesh Shah edited comment on YARN-5659 at 9/28/16 6:06 PM: [~sershe] Just annotate the functions that are only required by unit tests with @Private and @VisibleForTesting . In this case, the simplest approach would to use the above annotations for all the new functions that are added as part of this patch. was (Author: hitesh): [~sershe] Just annotate the functions that are only required by unit tests with @Private and @VisibleForTesting > getPathFromYarnURL should use standard methods > -- > > Key: YARN-5659 > URL: https://issues.apache.org/jira/browse/YARN-5659 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: YARN-5659.01.patch, YARN-5659.02.patch, > YARN-5659.03.patch, YARN-5659.04.patch, YARN-5659.04.patch, YARN-5659.patch > > > getPathFromYarnURL does some string shenanigans where standard ctors should > suffice. > There are also bugs in it e.g. passing an empty scheme to the URI ctor is > invalid, null should be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5659) getPathFromYarnURL should use standard methods
[ https://issues.apache.org/jira/browse/YARN-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530399#comment-15530399 ] Sergey Shelukhin commented on YARN-5659: Hmm.. which one should I add? I am not very familiar with approach in YARN. Can someone add those on commit? > getPathFromYarnURL should use standard methods > -- > > Key: YARN-5659 > URL: https://issues.apache.org/jira/browse/YARN-5659 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: YARN-5659.01.patch, YARN-5659.02.patch, > YARN-5659.03.patch, YARN-5659.04.patch, YARN-5659.04.patch, YARN-5659.patch > > > getPathFromYarnURL does some string shenanigans where standard ctors should > suffice. > There are also bugs in it e.g. passing an empty scheme to the URI ctor is > invalid, null should be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5685) Non-embedded HA failover is broken
Daniel Templeton created YARN-5685: -- Summary: Non-embedded HA failover is broken Key: YARN-5685 URL: https://issues.apache.org/jira/browse/YARN-5685 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.0.0-alpha1, 2.9.0 Reporter: Daniel Templeton Assignee: Daniel Templeton Priority: Critical YARN-4559 broke RM HA when embedded automatic failover is disabled. The {{ZKRMStateStore}} will now only start its monitoring thread when automatic failover not enabled (which is patently useless). I presume the intended change was to have the monitoring thread started when automatic failover is not *embedded*. If HA is enabled with automatic failover enabled and embedded failover disabled, all RMs all come up in standby state. To make one of them active, the {{--forcemanual}} flag must be used when manually triggering the state change. Should the active go down, the standby will not become active and must be manually transitioned with the {{--forcemanual}} flag. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2
[ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530324#comment-15530324 ] Joep Rottinghuis commented on YARN-4061: HBASE-12728 introduced the BufferedMutator. > [Fault tolerance] Fault tolerant writer for timeline v2 > --- > > Key: YARN-4061 > URL: https://issues.apache.org/jira/browse/YARN-4061 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Labels: YARN-5355 > Attachments: FaulttolerantwriterforTimelinev2.pdf > > > We need to build a timeline writer that can be resistant to backend storage > down time and timeline collector failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5599) Publish AM launch command to ATS
[ https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530315#comment-15530315 ] Rohith Sharma K S commented on YARN-5599: - updated branch-2 patch > Publish AM launch command to ATS > > > Key: YARN-5599 > URL: https://issues.apache.org/jira/browse/YARN-5599 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, > 0003-YARN-5599.patch, YARN-5599-branch-2.patch > > > To aid in debugging launch failures, it would be valuable to have an > application's launch script and logs posted to ATS. Because the > application's command line may contain private credentials or other secure > information, access to the data in ATS should be restricted to the job owner, > including the at-rest data. > Along with making the data available through ATS, the configuration parameter > introduced in YARN-5549 and the log line that it guards should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5599) Publish AM launch command to ATS
[ https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-5599: Attachment: YARN-5599-branch-2.patch > Publish AM launch command to ATS > > > Key: YARN-5599 > URL: https://issues.apache.org/jira/browse/YARN-5599 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, > 0003-YARN-5599.patch, YARN-5599-branch-2.patch > > > To aid in debugging launch failures, it would be valuable to have an > application's launch script and logs posted to ATS. Because the > application's command line may contain private credentials or other secure > information, access to the data in ATS should be restricted to the job owner, > including the at-rest data. > Along with making the data available through ATS, the configuration parameter > introduced in YARN-5549 and the log line that it guards should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4205) Add a service for monitoring application life time out
[ https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530187#comment-15530187 ] Hadoop QA commented on YARN-4205: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 49s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 14s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s {color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 569 unchanged - 3 fixed = 572 total (was 572) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 7s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s {color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s {color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 34m 52s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 66m 6s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830718/0007-YARN-4205.2.patch | | JIRA Issue | YARN-4205 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml | | uname | Linux 403272cbfc8b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / e19b37e | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apach
[jira] [Commented] (YARN-5676) Add a HashBasedRouterPolicy, that routes jobs based on queue name hash.
[ https://issues.apache.org/jira/browse/YARN-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530223#comment-15530223 ] Carlo Curino commented on YARN-5676: For very large queue structures, it is convenient to specify a policy for a subset of the queues (e.g., large/prod queues), and use something like the {{HashBasedRouterPolicy}} to automatically map all the small to subclusters. This is better than any of the randomized policies because the mapping between queue and sub-cluster tends to be more stable, hence there is some better chance for locality. I will post the patch soon (testing now). > Add a HashBasedRouterPolicy, that routes jobs based on queue name hash. > --- > > Key: YARN-5676 > URL: https://issues.apache.org/jira/browse/YARN-5676 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-2915 >Reporter: Carlo Curino >Assignee: Carlo Curino > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5600: --- Assignee: Miklos Szegedi (was: Daniel Templeton) > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3432) Cluster metrics have wrong Total Memory when there is reserved memory on CS
[ https://issues.apache.org/jira/browse/YARN-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530162#comment-15530162 ] Brahma Reddy Battula commented on YARN-3432: Oh sorry, missed this..didn't get time to work.. will try to update the unit testcase on next week. > Cluster metrics have wrong Total Memory when there is reserved memory on CS > --- > > Key: YARN-3432 > URL: https://issues.apache.org/jira/browse/YARN-3432 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler, resourcemanager >Affects Versions: 2.6.0 >Reporter: Thomas Graves >Assignee: Brahma Reddy Battula > Attachments: YARN-3432-002.patch, YARN-3432-003.patch, YARN-3432.patch > > > I noticed that when reservations happen when using the Capacity Scheduler, > the UI and web services report the wrong total memory. > For example. I have a 300GB of total memory in my cluster. I allocate 50 > and I reserve 10. The cluster metrics for total memory get reported as 290GB. > This was broken by https://issues.apache.org/jira/browse/YARN-656 so perhaps > there is a difference between fair scheduler and capacity scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5677) RM can be in active-active state for an extended period
[ https://issues.apache.org/jira/browse/YARN-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530124#comment-15530124 ] Daniel Templeton commented on YARN-5677: [~jianhe], is leader election on by default in Hadoop 3? I'd recommend it. Shall I roll that into the patch for this JIRA? > RM can be in active-active state for an extended period > --- > > Key: YARN-5677 > URL: https://issues.apache.org/jira/browse/YARN-5677 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.3, 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > > Both branch-2.8/trunk and branch-2.7 have issues when the active RM loses > contact with the ZK node(s). > In branch-2.7, the RM will retry the connection 1000 times by default. > Attempting to contact a node which cannot be reached is slow, which means the > active can take over an hour to realize it is no longer active. I clocked it > at about an hour and a half in my tests. The solution appears to be to add > some time awareness into the retry loop. > In branch-2.8/trunk, there is no maximum number of retries that I see. It > appears the connection will be retried forever, with the active never > figuring out it's no longer active. In my testing, the active-active state > lasted almost 2 hours with no sign of stopping before I killed it. The > solution appears to be to cap the number of retries or amount of time spent > retrying. > This issue is significant because of the asynchronous nature of job > submission. If the active doesn't know it's not active, it will buffer up > job submissions until it finally realizes it has become the standby. Then it > will fail all the job submissions in bulk. In high-volume workflows, that > behavior can create huge mass job failures. > This issue is also important because the node managers will not fail over to > the new active until the old active realizes it's the standby. Workloads > submitted after the old active loses contact with ZK will therefore fail to > be executed regardless of which RM the clients contact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5677) RM can be in active-active state for an extended period
[ https://issues.apache.org/jira/browse/YARN-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530118#comment-15530118 ] Daniel Templeton commented on YARN-5677: Just tested the leader election (with the right property enabled), and it works as advertised. The active becomes standby by the time the standby becomes active. > RM can be in active-active state for an extended period > --- > > Key: YARN-5677 > URL: https://issues.apache.org/jira/browse/YARN-5677 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.3, 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > > Both branch-2.8/trunk and branch-2.7 have issues when the active RM loses > contact with the ZK node(s). > In branch-2.7, the RM will retry the connection 1000 times by default. > Attempting to contact a node which cannot be reached is slow, which means the > active can take over an hour to realize it is no longer active. I clocked it > at about an hour and a half in my tests. The solution appears to be to add > some time awareness into the retry loop. > In branch-2.8/trunk, there is no maximum number of retries that I see. It > appears the connection will be retried forever, with the active never > figuring out it's no longer active. In my testing, the active-active state > lasted almost 2 hours with no sign of stopping before I killed it. The > solution appears to be to cap the number of retries or amount of time spent > retrying. > This issue is significant because of the asynchronous nature of job > submission. If the active doesn't know it's not active, it will buffer up > job submissions until it finally realizes it has become the standby. Then it > will fail all the job submissions in bulk. In high-volume workflows, that > behavior can create huge mass job failures. > This issue is also important because the node managers will not fail over to > the new active until the old active realizes it's the standby. Workloads > submitted after the old active loses contact with ZK will therefore fail to > be executed regardless of which RM the clients contact. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5145) [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR
[ https://issues.apache.org/jira/browse/YARN-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530061#comment-15530061 ] Sunil G commented on YARN-5145: --- Currently {{/logs}} has added as a new context to existing webserver in RM. Similarly we could try adding a new Context for new web UI. I am making some prototype for same, and will update status here. > [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR > - > > Key: YARN-5145 > URL: https://issues.apache.org/jira/browse/YARN-5145 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Kai Sasaki > Attachments: YARN-5145-YARN-3368.01.patch > > > Existing YARN UI configuration is under Hadoop package's directory: > $HADOOP_PREFIX/share/hadoop/yarn/webapps/, we should move it to > $HADOOP_CONF_DIR like other configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2
[ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529989#comment-15529989 ] Joep Rottinghuis commented on YARN-4061: I'm going to write up some requirements for what we'd want from a spooling BufferedMutator and open an HBase jira for the same. We get a type safe BufferedMutator from a shared connection in o.a.h.yarn.server.timelineservice.storage.common.BaseTable#getBufferedMutator In HBase the BufferedMutatorImpl is created up in HConnectionImplementation#getBufferedMutator, a static class inside ConnectionManager. It implements ClusterConnection which extends (the deprecated) HConnection, which in turn extends the Connection interface. While we can pass BufferedMutatorParams, but have no other way to inject our own implementation. This means we'll have to either wrap the return implementation and override and delegate to the methods, or have some modifications in HBase in order to be able to create a different implementation. BufferedMutatorParams has a (setter) method #listener which allows us to pass a BufferedMutator.ExceptionListener, this gives a glimmer of hope to be able to capture exceptions and possibly direct the BufferedMutator wrapper class to (temporarily) spool mutations to a FileSystem implementation. This could be HDFS, local filesystem, s3, gcs, or whatever the user wants to configure as a perfix path. > [Fault tolerance] Fault tolerant writer for timeline v2 > --- > > Key: YARN-4061 > URL: https://issues.apache.org/jira/browse/YARN-4061 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Labels: YARN-5355 > Attachments: FaulttolerantwriterforTimelinev2.pdf > > > We need to build a timeline writer that can be resistant to backend storage > down time and timeline collector failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4205) Add a service for monitoring application life time out
[ https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4205: Attachment: 0007-YARN-4205.2.patch Fixing TestPBImplRecords failure and some of the checkstyles > Add a service for monitoring application life time out > -- > > Key: YARN-4205 > URL: https://issues.apache.org/jira/browse/YARN-4205 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: nijel >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4205.patch, 0002-YARN-4205.patch, > 0003-YARN-4205.patch, 0004-YARN-4205.patch, 0005-YARN-4205.patch, > 0006-YARN-4205.patch, 0007-YARN-4205.1.patch, 0007-YARN-4205.2.patch, > 0007-YARN-4205.patch, YARN-4205_01.patch, YARN-4205_02.patch, > YARN-4205_03.patch > > > This JIRA intend to provide a lifetime monitor service. > The service will monitor the applications where the life time is configured. > If the application is running beyond the lifetime, it will be killed. > The lifetime will be considered from the submit time. > The thread monitoring interval is configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5684) testDecreaseAfterIncreaseWithAllocationExpiration fails intermittently
Rohith Sharma K S created YARN-5684: --- Summary: testDecreaseAfterIncreaseWithAllocationExpiration fails intermittently Key: YARN-5684 URL: https://issues.apache.org/jira/browse/YARN-5684 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Rohith Sharma K S Saw the following in a precommit: {code} org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer testDecreaseAfterIncreaseWithAllocationExpiration(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer) Time elapsed: 10.726 sec <<< FAILURE! java.lang.AssertionError: expected:<3> but was:<2> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer.testDecreaseAfterIncreaseWithAllocationExpiration(TestIncreaseAllocationExpirer.java:367) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4205) Add a service for monitoring application life time out
[ https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529649#comment-15529649 ] Hadoop QA commented on YARN-4205: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 58s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 31s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 44s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 12s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 7s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m 7s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 7s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 51s {color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 539 unchanged - 3 fixed = 559 total (was 542) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 51s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s {color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 35s {color} | {color:red} hadoop-yarn-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 48s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 74m 36s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.api.TestPBImplRecords | | | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer | | | hadoop.yarn.server.resourcemanager.TestRMAdminService | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830685/0007-YARN-4205.1.patch | | JIRA Issue | YARN-4205 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml | | uname | Linux 9550dff67904 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/L
[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529614#comment-15529614 ] Lei Guo commented on YARN-5683: --- This is related the constraint label YARN-3409, but different at detail level. This will rely on the constraint label for initial scheduling to find server with proper resource, and then node manager to help to set up proper environment for the workload. It's not just storage, we will also face similar issue on GPU or other hardware accelerator. > Support specifying storage type for per-application local dirs > -- > > Key: YARN-5683 > URL: https://issues.apache.org/jira/browse/YARN-5683 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Tao Yang > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5683-1.patch, flow_diagram_for_MapReduce.png > > > h3. Introduction > * Some applications of various frameworks (Flink, Spark and MapReduce etc) > using local storage (checkpoint, shuffle etc) might require high IO > performance. It's useful to allocate local directories to high performance > storage media for these applications on heterogeneous clusters. > * YARN does not distinguish different storage types and hence applications > cannot selectively use storage media with different performance > characteristics. Adding awareness of storage media can allow YARN to make > better decisions about the placement of local directories. > h3. Approach > * NodeManager will distinguish storage types for local directories. > ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration > should allow the cluster administrator to optionally specify the storage type > for each local directories. Example: > [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to > [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) > ** StorageType defines DISK/SSD storage types and takes DISK as the default > storage type. > ** StorageLocation separates storage type and directory path, used by > LocalDirAllocator to aware the types of local dirs, the default storage type > is DISK. > ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the > local directory of the specified storage type, and will fallback to not care > storage type if the requirement can not be satisfied. > ** Support for container related local/log directories by ContainerLaunch. > All application frameworks can set the environment variables > (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage > type of local/log directories. > * Allow specified storage type for various frameworks (Take MapReduce as an > example) > ** Add new configurations should allow application administrator to > optionally specify the storage type of local/log directories. (MapReduce add > configurations: mapreduce.job.local-storage-type and > mapreduce.job.log-storage-type) > ** Support for container work directories. Set the environment variables > includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations > above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce > should update YARNRunner and TaskAttemptImpl) > ** Add storage type prefix for request path to support for other local > directories of frameworks (such as shuffle directories for MapReduce). > (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to > support for output/work directories) > ** Flow diagram for MapReduce framework > !flow_diagram_for_MapReduce.png! > h3. Further Discussion > * The requirement of storage type for local/log directories may not be > satisfied on heterogeneous clusters. To achieve global optimum, scheduler > should aware and management disk resources to. > [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that > but seems not support multiple storage types, maybe we should do even more to > aware the storage type of disk resource? > * Node labels or node constraints can also make a higher chance to satisfy > the requirement of specified storage type. > * Fallback strategy still needs to be concerned. Certain applications might > not work well when the requirement of storage type is not satisfied. When > none of desired storage type disk are available, should container launching > be failed? let AM handle? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5599) Publish AM launch command to ATS
[ https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529549#comment-15529549 ] Varun Saxena commented on YARN-5599: [~rohithsharma] can you provide a branch-2 patch ? This would basically mean dropping ATSv2 changes. > Publish AM launch command to ATS > > > Key: YARN-5599 > URL: https://issues.apache.org/jira/browse/YARN-5599 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, > 0003-YARN-5599.patch > > > To aid in debugging launch failures, it would be valuable to have an > application's launch script and logs posted to ATS. Because the > application's command line may contain private credentials or other secure > information, access to the data in ATS should be restricted to the job owner, > including the at-rest data. > Along with making the data available through ATS, the configuration parameter > introduced in YARN-5549 and the log line that it guards should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4205) Add a service for monitoring application life time out
[ https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529513#comment-15529513 ] Hadoop QA commented on YARN-4205: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 35s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 57s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 48s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 46s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 4s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 17s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s {color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 18 new + 540 unchanged - 3 fixed = 558 total (was 543) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s {color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 36s {color} | {color:red} hadoop-yarn-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 37s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 76m 44s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.api.TestPBImplRecords | | | hadoop.yarn.server.resourcemanager.TestRMRestart | | | hadoop.yarn.server.resourcemanager.TestRMAdminService | | | hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830678/0007-YARN-4205.patch | | JIRA Issue | YARN-4205 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml | | uname | Linux f39ec40bf432 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016
[jira] [Updated] (YARN-4205) Add a service for monitoring application life time out
[ https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4205: Attachment: 0007-YARN-4205.1.patch I forgot to include test file. Updating new patch with Test file included. > Add a service for monitoring application life time out > -- > > Key: YARN-4205 > URL: https://issues.apache.org/jira/browse/YARN-4205 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: nijel >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4205.patch, 0002-YARN-4205.patch, > 0003-YARN-4205.patch, 0004-YARN-4205.patch, 0005-YARN-4205.patch, > 0006-YARN-4205.patch, 0007-YARN-4205.1.patch, 0007-YARN-4205.patch, > YARN-4205_01.patch, YARN-4205_02.patch, YARN-4205_03.patch > > > This JIRA intend to provide a lifetime monitor service. > The service will monitor the applications where the life time is configured. > If the application is running beyond the lifetime, it will be killed. > The lifetime will be considered from the submit time. > The thread monitoring interval is configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-3139) Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler
[ https://issues.apache.org/jira/browse/YARN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529409#comment-15529409 ] Jian He edited comment on YARN-3139 at 9/28/16 12:10 PM: - Patch looks good to me. bq. Instead of taking readLock, I think we can check if getNode(container.getNodeId()) == null, and stop if it is. In the latest patch, do you intend to not check "getNode(container.getNodeId()) == null" ? I think this is possible if a node is removed while the container on that is being released by AM. was (Author: jianhe): Patch looks good to me. bq. Instead of taking readLock, I think we can check if getNode(container.getNodeId()) == null, and stop if it is. In the latest patch, do you intend to not check "getNode(container.getNodeId()) == null" ? > Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler > -- > > Key: YARN-3139 > URL: https://issues.apache.org/jira/browse/YARN-3139 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-3139.0.patch, YARN-3139.1.patch, YARN-3139.2.patch, > YARN-3139.3.patch, YARN-3139.4.patch, YARN-3139.5.patch > > > Enhance locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler, as > mentioned in YARN-3091, a possible solution is using read/write lock. Other > fine-graind locks for specific purposes / bugs should be addressed in > separated tickets. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3139) Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler
[ https://issues.apache.org/jira/browse/YARN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529409#comment-15529409 ] Jian He commented on YARN-3139: --- Patch looks good to me. bq. Instead of taking readLock, I think we can check if getNode(container.getNodeId()) == null, and stop if it is. In the latest patch, do you intend to not check "getNode(container.getNodeId()) == null" ? > Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler > -- > > Key: YARN-3139 > URL: https://issues.apache.org/jira/browse/YARN-3139 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-3139.0.patch, YARN-3139.1.patch, YARN-3139.2.patch, > YARN-3139.3.patch, YARN-3139.4.patch, YARN-3139.5.patch > > > Enhance locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler, as > mentioned in YARN-3091, a possible solution is using read/write lock. Other > fine-graind locks for specific purposes / bugs should be addressed in > separated tickets. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4205) Add a service for monitoring application life time out
[ https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529380#comment-15529380 ] Jian He commented on YARN-4205: --- latest patch looks good to me > Add a service for monitoring application life time out > -- > > Key: YARN-4205 > URL: https://issues.apache.org/jira/browse/YARN-4205 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: nijel >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4205.patch, 0002-YARN-4205.patch, > 0003-YARN-4205.patch, 0004-YARN-4205.patch, 0005-YARN-4205.patch, > 0006-YARN-4205.patch, 0007-YARN-4205.patch, YARN-4205_01.patch, > YARN-4205_02.patch, YARN-4205_03.patch > > > This JIRA intend to provide a lifetime monitor service. > The service will monitor the applications where the life time is configured. > If the application is running beyond the lifetime, it will be killed. > The lifetime will be considered from the submit time. > The thread monitoring interval is configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4205) Add a service for monitoring application life time out
[ https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4205: Attachment: 0007-YARN-4205.patch Updated the patch with following major changes from previous one with more general way. # Moved RMAppLifetimeMonitor to separate package i.e *org.apache.hadoop.yarn.server.resourcemanager.rmapp.monitor* # Created RMAppToMonitor custom class to hold monitoring key. This consists of ApplicationId+ApplicationTimeoutType. This is essential because same application can have multiple timeouts. # Removed ApplicationTimeouts class. # In ApplicationSubmissionContext#setApplicationTimeouts is changed to Map. In future, if we want to support more timeouts like in Gour use case, we just need to add enum and provide corresponding implementation in RMAppImpl. > Add a service for monitoring application life time out > -- > > Key: YARN-4205 > URL: https://issues.apache.org/jira/browse/YARN-4205 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: nijel >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4205.patch, 0002-YARN-4205.patch, > 0003-YARN-4205.patch, 0004-YARN-4205.patch, 0005-YARN-4205.patch, > 0006-YARN-4205.patch, 0007-YARN-4205.patch, YARN-4205_01.patch, > YARN-4205_02.patch, YARN-4205_03.patch > > > This JIRA intend to provide a lifetime monitor service. > The service will monitor the applications where the life time is configured. > If the application is running beyond the lifetime, it will be killed. > The lifetime will be considered from the submit time. > The thread monitoring interval is configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-5683: --- Description: h3. Introduction * Some applications of various frameworks (Flink, Spark and MapReduce etc) using local storage (checkpoint, shuffle etc) might require high IO performance. It's useful to allocate local directories to high performance storage media for these applications on heterogeneous clusters. * YARN does not distinguish different storage types and hence applications cannot selectively use storage media with different performance characteristics. Adding awareness of storage media can allow YARN to make better decisions about the placement of local directories. h3. Approach * NodeManager will distinguish storage types for local directories. ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration should allow the cluster administrator to optionally specify the storage type for each local directories. Example: [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) ** StorageType defines DISK/SSD storage types and takes DISK as the default storage type. ** StorageLocation separates storage type and directory path, used by LocalDirAllocator to aware the types of local dirs, the default storage type is DISK. ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the local directory of the specified storage type, and will fallback to not care storage type if the requirement can not be satisfied. ** Support for container related local/log directories by ContainerLaunch. All application frameworks can set the environment variables (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage type of local/log directories. * Allow specified storage type for various frameworks (Take MapReduce as an example) ** Add new configurations should allow application administrator to optionally specify the storage type of local/log directories. (MapReduce add configurations: mapreduce.job.local-storage-type and mapreduce.job.log-storage-type) ** Support for container work directories. Set the environment variables includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce should update YARNRunner and TaskAttemptImpl) ** Add storage type prefix for request path to support for other local directories of frameworks (such as shuffle directories for MapReduce). (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to support for output/work directories) ** Flow diagram for MapReduce framework !flow_diagram_for_MapReduce.png! h3. Further Discussion * The requirement of storage type for local/log directories may not be satisfied on heterogeneous clusters. To achieve global optimum, scheduler should aware and management disk resources to. [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that but seems not support multiple storage types, maybe we should do even more to aware the storage type of disk resource? * Node labels or node constraints can also make a higher chance to satisfy the requirement of specified storage type. * Fallback strategy still needs to be concerned. Certain applications might not work well when the requirement of storage type is not satisfied. When none of desired storage type disk are available, should container launching be failed? let AM handle? was: h3. Introduction * Some applications of various frameworks (Flink, Spark and MapReduce etc) using local storage (checkpoint, shuffle etc) might require high IO performance. It's useful to allocate local directories to high performance storage media for these applications on heterogeneous clusters. * YARN does not distinguish different storage types and hence applications cannot selectively use storage media with different performance characteristics. Adding awareness of storage media can allow YARN to make better decisions about the placement of local/log directories with input from applications. An application can choose the desired storage media by configuration based on its performance and requirements. h3. Approach * NodeManager will distinguish storage types for local directories. ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration should allow the cluster administrator to optionally specify the storage type for each local directories. Example: [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) ** StorageType defines DISK/SSD storage types and takes DISK as the default storage type. ** StorageLocation separates storage type and directory path, used by LocalDirAllocator to aware the types of
[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-5683: --- Description: h3. Introduction * Some applications of various frameworks (Flink, Spark and MapReduce etc) using local storage (checkpoint, shuffle etc) might require high IO performance. It's useful to allocate local directories to high performance storage media for these applications on heterogeneous clusters. * YARN does not distinguish different storage types and hence applications cannot selectively use storage media with different performance characteristics. Adding awareness of storage media can allow YARN to make better decisions about the placement of local/log directories with input from applications. An application can choose the desired storage media by configuration based on its performance and requirements. h3. Approach * NodeManager will distinguish storage types for local directories. ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration should allow the cluster administrator to optionally specify the storage type for each local directories. Example: [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) ** StorageType defines DISK/SSD storage types and takes DISK as the default storage type. ** StorageLocation separates storage type and directory path, used by LocalDirAllocator to aware the types of local dirs, the default storage type is DISK. ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the local directory of the specified storage type, and will fallback to not care storage type if the requirement can not be satisfied. ** Support for container related local/log directories by ContainerLaunch. All application frameworks can set the environment variables (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage type of local/log directories. * Allow specified storage type for various frameworks (Take MapReduce as an example) ** Add new configurations should allow application administrator to optionally specify the storage type of local/log directories. (MapReduce add configurations: mapreduce.job.local-storage-type and mapreduce.job.log-storage-type) ** Support for container work directories. Set the environment variables includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce should update YARNRunner and TaskAttemptImpl) ** Add storage type prefix for request path to support for other local directories of frameworks (such as shuffle directories for MapReduce). (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to support for output/work directories) ** Flow diagram for MapReduce framework !flow_diagram_for_MapReduce.png! h3. Further Discussion * The requirement of storage type for local/log directories may not be satisfied on heterogeneous clusters. To achieve global optimum, scheduler should aware and management disk resources to. [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that but seems not support multiple storage types, maybe we should do even more to aware the storage type of disk resource? * Node labels or node constraints can also make a higher chance to satisfy the requirement of specified storage type. * Fallback strategy still needs to be concerned. Certain applications might not work well when the requirement of storage type is not satisfied. When none of desired storage type disk are available, should container launching be failed? let AM handle? was: h1. Introduction * Some applications of various frameworks (Flink, Spark and MapReduce etc) using local storage (checkpoint, shuffle etc) might require high IO performance. It's useful to allocate local directories to high performance storage media for these applications on heterogeneous clusters. * YARN does not distinguish different storage types and hence applications cannot selectively use storage media with different performance characteristics. Adding awareness of storage media can allow YARN to make better decisions about the placement of local/log directories with input from applications. An application can choose the desired storage media by configuration based on its performance and requirements. h1. Approach * NodeManager will distinguish storage types for local directories. ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration should allow the cluster administrator to optionally specify the storage type for each local directories. Example: [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) ** StorageType defines DISK/SSD storage types and takes
[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-5683: --- Attachment: flow_diagram_for_MapReduce.png > Support specifying storage type for per-application local dirs > -- > > Key: YARN-5683 > URL: https://issues.apache.org/jira/browse/YARN-5683 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Tao Yang > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5683-1.patch, flow_diagram_for_MapReduce.png > > > h1. Introduction > * Some applications of various frameworks (Flink, Spark and MapReduce etc) > using local storage (checkpoint, shuffle etc) might require high IO > performance. It's useful to allocate local directories to high performance > storage media for these applications on heterogeneous clusters. > * YARN does not distinguish different storage types and hence applications > cannot selectively use storage media with different performance > characteristics. Adding awareness of storage media can allow YARN to make > better decisions about the placement of local/log directories with input from > applications. An application can choose the desired storage media by > configuration based on its performance and requirements. > h1. Approach > * NodeManager will distinguish storage types for local directories. > ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration > should allow the cluster administrator to optionally specify the storage type > for each local directories. Example: > [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to > [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) > ** StorageType defines DISK/SSD storage types and takes DISK as the default > storage type. > ** StorageLocation separates storage type and directory path, used by > LocalDirAllocator to aware the types of local dirs, the default storage type > is DISK. > ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the > local directory of the specified storage type, and will fallback to not care > storage type if the requirement can not be satisfied. > ** Support for container related local/log directories by ContainerLaunch. > All application frameworks can set the environment variables > (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage > type of local/log directories. > * Allow specified storage type for various frameworks (Take MapReduce as an > example) > ** Add new configurations should allow application administrator to > optionally specify the storage type of local/log directories. (MapReduce add > configurations: mapreduce.job.local-storage-type and > mapreduce.job.log-storage-type) > ** Support for container work directories. Set the environment variables > includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations > above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce > should update YARNRunner and TaskAttemptImpl) > ** Add storage type prefix for request path to support for other local > directories of frameworks (such as shuffle directories for MapReduce). > (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to > support for output/work directories) > h1. Further Discussion > * The requirement of storage type for local/log directories may not be > satisfied on heterogeneous clusters. To achieve global optimum, scheduler > should aware and management disk resources to. > [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that > but seems not support multiple storage types, maybe we should do even more to > aware the storage type of disk resource? > * Node labels or node constraints can also make a higher chance to satisfy > the requirement of specified storage type. > * Fallback strategy still needs to be concerned. Certain applications might > not work well when the requirement of storage type is not satisfied. When > none of desired storage type disk are available, should container launching > be failed? let AM handle? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-5683: --- Attachment: YARN-5683-1.patch > Support specifying storage type for per-application local dirs > -- > > Key: YARN-5683 > URL: https://issues.apache.org/jira/browse/YARN-5683 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Tao Yang > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5683-1.patch > > > h1. Introduction > * Some applications of various frameworks (Flink, Spark and MapReduce etc) > using local storage (checkpoint, shuffle etc) might require high IO > performance. It's useful to allocate local directories to high performance > storage media for these applications on heterogeneous clusters. > * YARN does not distinguish different storage types and hence applications > cannot selectively use storage media with different performance > characteristics. Adding awareness of storage media can allow YARN to make > better decisions about the placement of local/log directories with input from > applications. An application can choose the desired storage media by > configuration based on its performance and requirements. > h1. Approach > * NodeManager will distinguish storage types for local directories. > ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration > should allow the cluster administrator to optionally specify the storage type > for each local directories. Example: > [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to > [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) > ** StorageType defines DISK/SSD storage types and takes DISK as the default > storage type. > ** StorageLocation separates storage type and directory path, used by > LocalDirAllocator to aware the types of local dirs, the default storage type > is DISK. > ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the > local directory of the specified storage type, and will fallback to not care > storage type if the requirement can not be satisfied. > ** Support for container related local/log directories by ContainerLaunch. > All application frameworks can set the environment variables > (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage > type of local/log directories. > * Allow specified storage type for various frameworks (Take MapReduce as an > example) > ** Add new configurations should allow application administrator to > optionally specify the storage type of local/log directories. (MapReduce add > configurations: mapreduce.job.local-storage-type and > mapreduce.job.log-storage-type) > ** Support for container work directories. Set the environment variables > includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations > above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce > should update YARNRunner and TaskAttemptImpl) > ** Add storage type prefix for request path to support for other local > directories of frameworks (such as shuffle directories for MapReduce). > (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to > support for output/work directories) > h1. Further Discussion > * The requirement of storage type for local/log directories may not be > satisfied on heterogeneous clusters. To achieve global optimum, scheduler > should aware and management disk resources to. > [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that > but seems not support multiple storage types, maybe we should do even more to > aware the storage type of disk resource? > * Node labels or node constraints can also make a higher chance to satisfy > the requirement of specified storage type. > * Fallback strategy still needs to be concerned. Certain applications might > not work well when the requirement of storage type is not satisfied. When > none of desired storage type disk are available, should container launching > be failed? let AM handle? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs
[ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-5683: --- Description: h1. Introduction * Some applications of various frameworks (Flink, Spark and MapReduce etc) using local storage (checkpoint, shuffle etc) might require high IO performance. It's useful to allocate local directories to high performance storage media for these applications on heterogeneous clusters. * YARN does not distinguish different storage types and hence applications cannot selectively use storage media with different performance characteristics. Adding awareness of storage media can allow YARN to make better decisions about the placement of local/log directories with input from applications. An application can choose the desired storage media by configuration based on its performance and requirements. h1. Approach * NodeManager will distinguish storage types for local directories. ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration should allow the cluster administrator to optionally specify the storage type for each local directories. Example: [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) ** StorageType defines DISK/SSD storage types and takes DISK as the default storage type. ** StorageLocation separates storage type and directory path, used by LocalDirAllocator to aware the types of local dirs, the default storage type is DISK. ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the local directory of the specified storage type, and will fallback to not care storage type if the requirement can not be satisfied. ** Support for container related local/log directories by ContainerLaunch. All application frameworks can set the environment variables (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage type of local/log directories. * Allow specified storage type for various frameworks (Take MapReduce as an example) ** Add new configurations should allow application administrator to optionally specify the storage type of local/log directories. (MapReduce add configurations: mapreduce.job.local-storage-type and mapreduce.job.log-storage-type) ** Support for container work directories. Set the environment variables includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce should update YARNRunner and TaskAttemptImpl) ** Add storage type prefix for request path to support for other local directories of frameworks (such as shuffle directories for MapReduce). (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to support for output/work directories) h1. Further Discussion * The requirement of storage type for local/log directories may not be satisfied on heterogeneous clusters. To achieve global optimum, scheduler should aware and management disk resources to. [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that but seems not support multiple storage types, maybe we should do even more to aware the storage type of disk resource? * Node labels or node constraints can also make a higher chance to satisfy the requirement of specified storage type. * Fallback strategy still needs to be concerned. Certain applications might not work well when the requirement of storage type is not satisfied. When none of desired storage type disk are available, should container launching be failed? let AM handle? was: # Introduction * Some applications of various frameworks (Flink, Spark and MapReduce etc) using local storage (checkpoint, shuffle etc) might require high IO performance. It's useful to allocate local directories to high performance storage media for these applications on heterogeneous clusters. * YARN does not distinguish different storage types and hence applications cannot selectively use storage media with different performance characteristics. Adding awareness of storage media can allow YARN to make better decisions about the placement of local/log directories with input from applications. An application can choose the desired storage media by configuration based on its performance and requirements. # Approach * NodeManager will distinguish storage types for local directories. ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration should allow the cluster administrator to optionally specify the storage type for each local directories. Example: [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) ** StorageType defines DISK/SSD storage types and takes DISK as the default storage type. ** StorageLocation separates storage type a
[jira] [Created] (YARN-5683) Support specifying storage type for per-application local dirs
Tao Yang created YARN-5683: -- Summary: Support specifying storage type for per-application local dirs Key: YARN-5683 URL: https://issues.apache.org/jira/browse/YARN-5683 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager Affects Versions: 3.0.0-alpha2 Reporter: Tao Yang Fix For: 3.0.0-alpha2 # Introduction * Some applications of various frameworks (Flink, Spark and MapReduce etc) using local storage (checkpoint, shuffle etc) might require high IO performance. It's useful to allocate local directories to high performance storage media for these applications on heterogeneous clusters. * YARN does not distinguish different storage types and hence applications cannot selectively use storage media with different performance characteristics. Adding awareness of storage media can allow YARN to make better decisions about the placement of local/log directories with input from applications. An application can choose the desired storage media by configuration based on its performance and requirements. # Approach * NodeManager will distinguish storage types for local directories. ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration should allow the cluster administrator to optionally specify the storage type for each local directories. Example: [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir) ** StorageType defines DISK/SSD storage types and takes DISK as the default storage type. ** StorageLocation separates storage type and directory path, used by LocalDirAllocator to aware the types of local dirs, the default storage type is DISK. ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the local directory of the specified storage type, and will fallback to not care storage type if the requirement can not be satisfied. ** Support for container related local/log directories by ContainerLaunch. All application frameworks can set the environment variables (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage type of local/log directories. * Allow specified storage type for various frameworks (Take MapReduce as an example) ** Add new configurations should allow application administrator to optionally specify the storage type of local/log directories. (MapReduce add configurations: mapreduce.job.local-storage-type and mapreduce.job.log-storage-type) ** Support for container work directories. Set the environment variables includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce should update YARNRunner and TaskAttemptImpl) ** Add storage type prefix for request path to support for other local directories of frameworks (such as shuffle directories for MapReduce). (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to support for output/work directories) # Further Discussion * The requirement of storage type for local/log directories may not be satisfied on heterogeneous clusters. To achieve global optimum, scheduler should aware and management disk resources to. [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that but seems not support multiple storage types, maybe we should do even more to aware the storage type of disk resource? * Node labels or node constraints can also make a higher chance to satisfy the requirement of specified storage type. * Fallback strategy still needs to be concerned. Certain applications might not work well when the requirement of storage type is not satisfied. When none of desired storage type disk are available, should container launching be failed? let AM handle? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5662) Provide an option to enable ContainerMonitor
[ https://issues.apache.org/jira/browse/YARN-5662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529266#comment-15529266 ] Hudson commented on YARN-5662: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10506 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10506/]) YARN-5662. Provide an option to enable ContainerMonitor. Contributed by (vvasudev: rev bc2656f09f857fdbc39da6b060cee453d2dec4dc) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerStatusPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > Provide an option to enable ContainerMonitor > - > > Key: YARN-5662 > URL: https://issues.apache.org/jira/browse/YARN-5662 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5662.1.patch, YARN-5662.2.patch, YARN-5662.3.patch > > > Currently, if vmem/pmem check is not enabled, ContainerMonitor would not run. > In certain cases, ContainerMonitor also needs to run to monitor things like > container-metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5599) Publish AM launch command to ATS
[ https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529267#comment-15529267 ] Hudson commented on YARN-5599: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10506 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10506/]) YARN-5599. Publish AM launch command to ATS (Rohith Sharma K S via Varun (varunsaxena: rev 9b0fd01d2ee002ac4c30c2862e18ca8f1626fa8d) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisherForV2.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TimelineServiceV1Publisher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TimelineServiceV2Publisher.java > Publish AM launch command to ATS > > > Key: YARN-5599 > URL: https://issues.apache.org/jira/browse/YARN-5599 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, > 0003-YARN-5599.patch > > > To aid in debugging launch failures, it would be valuable to have an > application's launch script and logs posted to ATS. Because the > application's command line may contain private credentials or other secure > information, access to the data in ATS should be restricted to the job owner, > including the at-rest data. > Along with making the data available through ATS, the configuration parameter > introduced in YARN-5549 and the log line that it guards should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4855) Should check if node exists when replace nodelabels
[ https://issues.apache.org/jira/browse/YARN-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529000#comment-15529000 ] Naganarasimha G R commented on YARN-4855: - bq. so it will be better to not add the hyphen-connected option. agree... bq. will strongly suggest to use the cliParser, but that can come in a separated patch. Ok In that case lets try to target this at the earliest and will not commit this patch to 2.8 until the followup jira comes in. [~Tao Jie], hope you can quickly create a patch to modify the option to "failOnUnknownNodes" and raise a new jira to use CliParser's options where ever required. > Should check if node exists when replace nodelabels > --- > > Key: YARN-4855 > URL: https://issues.apache.org/jira/browse/YARN-4855 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Tao Jie >Assignee: Tao Jie >Priority: Minor > Attachments: YARN-4855.001.patch, YARN-4855.002.patch, > YARN-4855.003.patch, YARN-4855.004.patch, YARN-4855.005.patch, > YARN-4855.006.patch, YARN-4855.007.patch, YARN-4855.008.patch, > YARN-4855.009.patch, YARN-4855.010.patch, YARN-4855.011.patch, > YARN-4855.012.patch > > > Today when we add nodelabels to nodes, it would succeed even if nodes are not > existing NodeManger in cluster without any message. > It could be like this: > When we use *yarn rmadmin -replaceLabelsOnNode --fail-on-unkown-nodes > "node1=label1"* , it would be denied if node is unknown. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5682) [YARN-3368] Fix maven build to keep all generated or downloaded files in target folder
[ https://issues.apache.org/jira/browse/YARN-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15528995#comment-15528995 ] Sunil G commented on YARN-5682: --- Hi [~wangda] I ran {{mvn install -Pyarn-ui}} with and w/o {{mvn clean}} after applying this patch. i think we are taking more time now. {noformat} mvn clean; mvn install -Pyarn-ui [INFO] Apache Hadoop YARN UI .. SUCCESS [01:56 min] mvn install -Pyarn-ui [INFO] Apache Hadoop YARN UI .. SUCCESS [01:51 min] {noformat} I tested {{mvn install -Pyarn-ui}} w/o this patch. It is taking lesser time. I think we need not have to clean temp dir's. {noformat} [INFO] Total time: 29.515 s {noformat} > [YARN-3368] Fix maven build to keep all generated or downloaded files in > target folder > -- > > Key: YARN-5682 > URL: https://issues.apache.org/jira/browse/YARN-5682 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5682-YARN-3368.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-4743) ResourceManager crash because TimSort
[ https://issues.apache.org/jira/browse/YARN-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15528957#comment-15528957 ] Yufei Gu edited comment on YARN-4743 at 9/28/16 9:06 AM: - Hi [~gzh1992n], thanks for working on this. Some thoughts about the patch. Both 0.5(weight value less than 1.0) or 0.0 are valid value for weights in fair scheduler. Once use case of zero-weight would be that user uses the zero-weight queue to run jobs when there is no jobs for other non-zero-weight queues. So it make no sense to me to enforce weight larger than 1.0. If NaN affects the transitive, we can avoid NaN by other ways. For example, if the first weight is 0.0 and the second is bigger than 0.0, obviously, the second one is needier than the first one. was (Author: yufeigu): Hi [~gzh1992n], thanks for working on this. Some thoughts about the patch. Both 0.5(weight value less than 1.0) or 0.0 are valid value for weights in fair scheduler. Once use case of zero-weight would be that user uses the zero-weight queue to run jobs when there is no jobs for other non-zero-weight queues. So it make no sense to me to enforce weight larger than 1.0. > ResourceManager crash because TimSort > - > > Key: YARN-4743 > URL: https://issues.apache.org/jira/browse/YARN-4743 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.4 >Reporter: Zephyr Guo > Fix For: 3.0.0-alpha1 > > Attachments: YARN-4743-v1.patch, YARN-CDH5.4.7.patch, timsort.log > > > {code} > 2016-02-26 14:08:50,821 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: Comparison method violates its general > contract! > at java.util.TimSort.mergeHi(TimSort.java:868) > at java.util.TimSort.mergeAt(TimSort.java:485) > at java.util.TimSort.mergeCollapse(TimSort.java:410) > at java.util.TimSort.sort(TimSort.java:214) > at java.util.TimSort.sort(TimSort.java:173) > at java.util.Arrays.sort(Arrays.java:659) > at java.util.Collections.sort(Collections.java:217) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:316) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:240) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1091) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:989) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1185) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:684) > at java.lang.Thread.run(Thread.java:745) > 2016-02-26 14:08:50,822 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, bbye.. > {code} > Actually, this issue found in 2.6.0-cdh5.4.7. > I think the cause is that we modify {{Resouce}} while we are sorting > {{runnableApps}}. > {code:title=FSLeafQueue.java} > Comparator comparator = policy.getComparator(); > writeLock.lock(); > try { > Collections.sort(runnableApps, comparator); > } finally { > writeLock.unlock(); > } > readLock.lock(); > {code} > {code:title=FairShareComparator} > public int compare(Schedulable s1, Schedulable s2) { > .. > s1.getResourceUsage(), minShare1); > boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null, > s2.getResourceUsage(), minShare2); > minShareRatio1 = (double) s1.getResourceUsage().getMemory() > / Resources.max(RESOURCE_CALCULATOR, null, minShare1, > ONE).getMemory(); > minShareRatio2 = (double) s2.getResourceUsage().getMemory() > / Resources.max(RESOURCE_CALCULATOR, null, minShare2, > ONE).getMemory(); > .. > {code} > {{getResourceUsage}} will return current Resource. The current Resource is > unstable. > {code:title=FSAppAttempt.java} > @Override > public Resource getResourceUsage() { > // Here the getPreemptedResources() always return zero, except in > // a preemption round > return Resources.subtract(getCurrentConsumption(), > getPreemptedResources()); > } > {code} > {code:title=SchedulerApplicationAttempt} > public Resource getCurrentConsumption() { > return currentConsumption; > } > // This method
[jira] [Commented] (YARN-4743) ResourceManager crash because TimSort
[ https://issues.apache.org/jira/browse/YARN-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15528957#comment-15528957 ] Yufei Gu commented on YARN-4743: Hi [~gzh1992n], thanks for working on this. Some thoughts about the patch. Both 0.5(weight value less than 1.0) or 0.0 are valid value for weights in fair scheduler. Once use case of zero-weight would be that user uses the zero-weight queue to run jobs when there is no jobs for other non-zero-weight queues. So it make no sense to me to enforce weight larger than 1.0. > ResourceManager crash because TimSort > - > > Key: YARN-4743 > URL: https://issues.apache.org/jira/browse/YARN-4743 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.4 >Reporter: Zephyr Guo > Fix For: 3.0.0-alpha1 > > Attachments: YARN-4743-v1.patch, YARN-CDH5.4.7.patch, timsort.log > > > {code} > 2016-02-26 14:08:50,821 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: Comparison method violates its general > contract! > at java.util.TimSort.mergeHi(TimSort.java:868) > at java.util.TimSort.mergeAt(TimSort.java:485) > at java.util.TimSort.mergeCollapse(TimSort.java:410) > at java.util.TimSort.sort(TimSort.java:214) > at java.util.TimSort.sort(TimSort.java:173) > at java.util.Arrays.sort(Arrays.java:659) > at java.util.Collections.sort(Collections.java:217) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:316) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:240) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1091) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:989) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1185) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:684) > at java.lang.Thread.run(Thread.java:745) > 2016-02-26 14:08:50,822 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, bbye.. > {code} > Actually, this issue found in 2.6.0-cdh5.4.7. > I think the cause is that we modify {{Resouce}} while we are sorting > {{runnableApps}}. > {code:title=FSLeafQueue.java} > Comparator comparator = policy.getComparator(); > writeLock.lock(); > try { > Collections.sort(runnableApps, comparator); > } finally { > writeLock.unlock(); > } > readLock.lock(); > {code} > {code:title=FairShareComparator} > public int compare(Schedulable s1, Schedulable s2) { > .. > s1.getResourceUsage(), minShare1); > boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null, > s2.getResourceUsage(), minShare2); > minShareRatio1 = (double) s1.getResourceUsage().getMemory() > / Resources.max(RESOURCE_CALCULATOR, null, minShare1, > ONE).getMemory(); > minShareRatio2 = (double) s2.getResourceUsage().getMemory() > / Resources.max(RESOURCE_CALCULATOR, null, minShare2, > ONE).getMemory(); > .. > {code} > {{getResourceUsage}} will return current Resource. The current Resource is > unstable. > {code:title=FSAppAttempt.java} > @Override > public Resource getResourceUsage() { > // Here the getPreemptedResources() always return zero, except in > // a preemption round > return Resources.subtract(getCurrentConsumption(), > getPreemptedResources()); > } > {code} > {code:title=SchedulerApplicationAttempt} > public Resource getCurrentConsumption() { > return currentConsumption; > } > // This method may modify current Resource. > public synchronized void recoverContainer(RMContainer rmContainer) { > .. > Resources.addTo(currentConsumption, rmContainer.getContainer() > .getResource()); > .. > } > {code} > I suggest that use stable Resource in comparator. > Is there something i think wrong? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5682) [YARN-3368] Fix maven build to keep all generated or downloaded files in target folder
[ https://issues.apache.org/jira/browse/YARN-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15528887#comment-15528887 ] Hadoop QA commented on YARN-5682: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m 58s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m 9s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 45s {color} | {color:green} YARN-3368 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s {color} | {color:green} YARN-3368 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 9s {color} | {color:green} YARN-3368 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 42s {color} | {color:green} YARN-3368 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s {color} | {color:green} YARN-3368 passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 4s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 25s {color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 7s {color} | {color:green} hadoop-yarn-ui in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 59m 4s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer | | | hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices | | | hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9baccb9 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830637/YARN-5682-YARN-3368.001.patch | | JIRA Issue | YARN-5682 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml | | uname | Linux a6bb981b5b67 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | YARN-3368 / 9ef8291 | | Default Java | 1.8.0_101 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/13234/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt | | unit test logs | https://builds.apache.org/job/PreCommit-YARN-Build/13234/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13234/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: hadoop-yarn-project/hadoop-yarn | | Console output | https://b
[jira] [Commented] (YARN-5554) MoveApplicationAcrossQueues does not check user permission on the target queue
[ https://issues.apache.org/jira/browse/YARN-5554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15528729#comment-15528729 ] Yufei Gu commented on YARN-5554: The latest patch looks good to me. +1(non-binding). > MoveApplicationAcrossQueues does not check user permission on the target queue > -- > > Key: YARN-5554 > URL: https://issues.apache.org/jira/browse/YARN-5554 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Wilfred Spiegelenburg > Attachments: YARN-5554.2.patch, YARN-5554.3.patch, YARN-5554.4.patch, > YARN-5554.5.patch, YARN-5554.6.patch, YARN-5554.7.patch > > > moveApplicationAcrossQueues operation currently does not check user > permission on the target queue. This incorrectly allows one user to move > his/her own applications to a queue that the user has no access to -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5599) Publish AM launch command to ATS
[ https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15528688#comment-15528688 ] Varun Saxena commented on YARN-5599: +1. Will commit shortly. > Publish AM launch command to ATS > > > Key: YARN-5599 > URL: https://issues.apache.org/jira/browse/YARN-5599 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, > 0003-YARN-5599.patch > > > To aid in debugging launch failures, it would be valuable to have an > application's launch script and logs posted to ATS. Because the > application's command line may contain private credentials or other secure > information, access to the data in ATS should be restricted to the job owner, > including the at-rest data. > Along with making the data available through ATS, the configuration parameter > introduced in YARN-5549 and the log line that it guards should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5599) Publish AM launch command to ATS
[ https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5599: --- Summary: Publish AM launch command to ATS (was: Post AM launcher artifacts to ATS) > Publish AM launch command to ATS > > > Key: YARN-5599 > URL: https://issues.apache.org/jira/browse/YARN-5599 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, > 0003-YARN-5599.patch > > > To aid in debugging launch failures, it would be valuable to have an > application's launch script and logs posted to ATS. Because the > application's command line may contain private credentials or other secure > information, access to the data in ATS should be restricted to the job owner, > including the at-rest data. > Along with making the data available through ATS, the configuration parameter > introduced in YARN-5549 and the log line that it guards should be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org