[jira] [Commented] (YARN-4806) Don't run TestZKRMStateStorePerf as part of the regular unittests
[ https://issues.apache.org/jira/browse/YARN-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407255#comment-15407255 ] Park Kyeong Hee commented on YARN-4806: --- Hi [~kasha] !! I'm newbie in here. And I think this issue is good try for me. So here is some question I have. It's possible that I add some codes on hadoop-project's pom.xml. (or restricted under hadoop-yarn project.) Is it OK that added profile may test only TestZKRMStateStorePerf at this time. > Don't run TestZKRMStateStorePerf as part of the regular unittests > - > > Key: YARN-4806 > URL: https://issues.apache.org/jira/browse/YARN-4806 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Karthik Kambatla > Labels: newbie, unit-test > > TestZKRMStateStorePerf takes about 5 minutes to run, thus adding 10 minutes > to the precommit build. We should run these on a separate profile. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[ https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407212#comment-15407212 ] Sunil G commented on YARN-5333: --- Hi [~jianhe] and [~rohithsharma] [~hex108] bq.I think the ActiveStandbyElector will handle reJoin automatically if exception thrown from the transtionToActive method.We can use the fail fast config for this scenario. I have another view here. If {{refreshAll}} fails due to a corrupted capacity-scheduler.xml file when a transitionToActive is happening (IOException will come from *refreshQeues*) as mentioned in YARN-3893, I think we need to fail-fast. Yes, it will be harsh. But we can avoid a scenario like both RMs in standby (and both RMs will switchover continuously) if I am not wrong. pls correct me if I understood this case wrong. This will happen if config is given not-to fail fast. pls share your thoughts. > Some recovered apps are put into default queue when RM HA > - > > Key: YARN-5333 > URL: https://issues.apache.org/jira/browse/YARN-5333 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: YARN-5333.01.patch, YARN-5333.02.patch, > YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch, > YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch, YARN-5333.09.patch > > > Enable RM HA and use FairScheduler, > {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false, > {{yarn.scheduler.fair.user-as-default-queue}} is set to false. > Reproduce steps: > 1. Start two RMs. > 2. After RMs are running, change both RM's file > {{etc/hadoop/fair-scheduler.xml}}, then add some queues. > 3. Submit some apps to the new added queues. > 4. Stop the active RM, then the standby RM will transit to active and recover > apps. > However the new active RM will put recovered apps into default queue because > it might have not loaded the new {{fair-scheduler.xml}}. We need call > {{initScheduler}} before start active services or bring {{refreshAll()}} in > front of {{rm.transitionToActive()}}. *It seems it is also important for > other scheduler*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5334) [YARN-3368] Introduce REFRESH button in various UI pages
[ https://issues.apache.org/jira/browse/YARN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407197#comment-15407197 ] Sunil G commented on YARN-5334: --- Hi [~leftnoteasy], For points 2~4, its same as point 1. For pages like cluster-overview or yarn-apps, we have to unload the model and query/find it again to refresh. The items which I have mentioned in the comments were queried without unloading. So we may not get fresh data in. Tests may seem fine. [~Sreenath], could you also please confirm the same. > [YARN-3368] Introduce REFRESH button in various UI pages > > > Key: YARN-5334 > URL: https://issues.apache.org/jira/browse/YARN-5334 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Sunil G >Assignee: Sreenath Somarajapuram > Attachments: YARN-5334-YARN-3368-0001.patch > > > It will be better if we have a common Refresh button in all pages to get the > latest information in all tables such as apps/nodes/queue etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5333) Some recovered apps are put into default queue when RM HA
[ https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Gong updated YARN-5333: --- Attachment: YARN-5333.09.patch > Some recovered apps are put into default queue when RM HA > - > > Key: YARN-5333 > URL: https://issues.apache.org/jira/browse/YARN-5333 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: YARN-5333.01.patch, YARN-5333.02.patch, > YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch, > YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch, YARN-5333.09.patch > > > Enable RM HA and use FairScheduler, > {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false, > {{yarn.scheduler.fair.user-as-default-queue}} is set to false. > Reproduce steps: > 1. Start two RMs. > 2. After RMs are running, change both RM's file > {{etc/hadoop/fair-scheduler.xml}}, then add some queues. > 3. Submit some apps to the new added queues. > 4. Stop the active RM, then the standby RM will transit to active and recover > apps. > However the new active RM will put recovered apps into default queue because > it might have not loaded the new {{fair-scheduler.xml}}. We need call > {{initScheduler}} before start active services or bring {{refreshAll()}} in > front of {{rm.transitionToActive()}}. *It seems it is also important for > other scheduler*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[ https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407189#comment-15407189 ] Jun Gong commented on YARN-5333: Attach a new patch 09.patch. Rename {{refreshXXXWithoutCheck}} to {{refreshXXX}}. Add {{checkAcls("refreshAll")}} at the beginning of {{refreshAll()}}, then we could check user's ACL. > Some recovered apps are put into default queue when RM HA > - > > Key: YARN-5333 > URL: https://issues.apache.org/jira/browse/YARN-5333 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: YARN-5333.01.patch, YARN-5333.02.patch, > YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch, > YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch > > > Enable RM HA and use FairScheduler, > {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false, > {{yarn.scheduler.fair.user-as-default-queue}} is set to false. > Reproduce steps: > 1. Start two RMs. > 2. After RMs are running, change both RM's file > {{etc/hadoop/fair-scheduler.xml}}, then add some queues. > 3. Submit some apps to the new added queues. > 4. Stop the active RM, then the standby RM will transit to active and recover > apps. > However the new active RM will put recovered apps into default queue because > it might have not loaded the new {{fair-scheduler.xml}}. We need call > {{initScheduler}} before start active services or bring {{refreshAll()}} in > front of {{rm.transitionToActive()}}. *It seems it is also important for > other scheduler*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5079) [Umbrella] Native YARN framework layer for services and beyond
[ https://issues.apache.org/jira/browse/YARN-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407057#comment-15407057 ] Jian He commented on YARN-5079: --- Hi [~asuresh], Short answer is, agent in Slider will be thrown away once merged in YARN. NodeManager will behave like the agent for upgrade, reconfigure etc. e.g. YARN-4726 will do most of the work. Of course, startContainer call in slider AM will be changed to start the real container, instead of the slider agent. For SLIDER-787, probably the client-facing code will be preserved, the internal logic will be changed to take advantage of YARN-4726. > [Umbrella] Native YARN framework layer for services and beyond > -- > > Key: YARN-5079 > URL: https://issues.apache.org/jira/browse/YARN-5079 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > > (See overview doc at YARN-4692, modifying and copy-pasting some of the > relevant pieces and sub-section 3.3.1 to track the specific sub-item.) > (This is a companion to YARN-4793 in our effort to simplify the entire story, > but focusing on APIs) > So far, YARN by design has restricted itself to having a very low-level API > that can support any type of application. Frameworks like Apache Hadoop > MapReduce, Apache Tez, Apache Spark, Apache REEF, Apache Twill, Apache Helix > and others ended up exposing higher level APIs that end-users can directly > leverage to build their applications on top of YARN. On the services side, > Apache Slider has done something similar. > With our current attention on making services first-class and simplified, > it's time to take a fresh look at how we can make Apache Hadoop YARN support > services well out of the box. Beyond the functionality that I outlined in the > previous sections in the doc on how NodeManagers can be enhanced to help > services, the biggest missing piece is the framework itself. There is a lot > of very important functionality that a services' framework can own together > with YARN in executing services end-to-end. > In this JIRA I propose we look at having a native Apache Hadoop framework for > running services natively on YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5449) nodemanager process is hung, and lost from resourcemanager
[ https://issues.apache.org/jira/browse/YARN-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401469#comment-15401469 ] mai shurong edited comment on YARN-5449 at 8/4/16 2:57 AM: --- Sorry, I had added my description of this issue when I created it, but was not submitted to jira by some problems. I would add description as soon as possible. was (Author: shurong.mai): Sorry, I had added my description of this issue when I created it, bu was not submitted to jira by some problems. I would add description as soon as possible. > nodemanager process is hung, and lost from resourcemanager > -- > > Key: YARN-5449 > URL: https://issues.apache.org/jira/browse/YARN-5449 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.2.0 > Environment: The os version is 2.6.32-573.8.1.el6.x86_64 GNU/Linux > The java version is jdk1.7.0_45 > The hadoop version is hadoop-2.2.0 >Reporter: mai shurong > > The nodemanager process is hung(is not dead), and lost from resourcemanager. > The nodemanager's log is stopped from printing. > The used cpu of nodemanager process is very low(nearly 0%). > GC of nodemanager jvm process is stopped, and the result of jstat(jstat > -gccause pid 1000 100) is as follows: > S0 S1 E O P YGC YGCTFGCFGCT GCT > LGCC GCC > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > 0.00 100.00 95.06 24.08 30.46 3274 623.437 75.899 629.335 No > GCG1 Evacuation Pause > The nodemanager jvm process is also accur this problem using CMS garbage > collector or g1 garbage collector. > The parameters of CMS garbage collector are as following: > -Xmx4096m -Xmn1024m -XX:PermSize=128m -XX:MaxPermSize=128m > -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:ConcGCThreads=4 > -XX:+UseCMSCom pactAtFullCollection -XX:CMSFullGCsBeforeCompaction=8 > -XX:ParallelGCThreads=4 -XX:CMSInitiatingOccupancyFraction=70 > The parameters of g1 garbage collector are as following: > -Xmx8g -Xms8g -XX:PermSize=128m -XX:MaxPermSize=128m -XX:+UseG1GC > -XX:MaxGCPauseMillis=1000 -XX:G1ReservePercent=30 > -XX:InitiatingHeapOccupancyPercent=45 -XX:ConcGCThreads=4 > -XX:+PrintAdaptiveSizePolicy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[ https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407041#comment-15407041 ] Jun Gong commented on YARN-5333: Thanks [~rohithsharma] for the review. bq. refreshXXXWithoutCheck does not looks meaning full method name. I think common general pattern can be followed like below. refreshXXXWithouCheck means that there is no check for refreshXXX. If refreshXXX is acceptable, I'd like to change it. bq. One of my major concern after seeing patch is skipping checkACL which used to verify user for every transition-to-active. But now it is skipped. I ignored it... It seems that we need add checkACL. How about adding it in {{refreshAll}}? {code} refreshAll () { checkACL("XXX"); refreshXXX(); ... } {code} bq. Test failure is related to patch change. I think this test can be removed only since behavior is changed after this patch. Yes, it is related, I fixed it in the patch 07.patch. The test case seems useful for testing the case that {{refreshAll}} failed. Maybe we could keep it? > Some recovered apps are put into default queue when RM HA > - > > Key: YARN-5333 > URL: https://issues.apache.org/jira/browse/YARN-5333 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: YARN-5333.01.patch, YARN-5333.02.patch, > YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch, > YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch > > > Enable RM HA and use FairScheduler, > {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false, > {{yarn.scheduler.fair.user-as-default-queue}} is set to false. > Reproduce steps: > 1. Start two RMs. > 2. After RMs are running, change both RM's file > {{etc/hadoop/fair-scheduler.xml}}, then add some queues. > 3. Submit some apps to the new added queues. > 4. Stop the active RM, then the standby RM will transit to active and recover > apps. > However the new active RM will put recovered apps into default queue because > it might have not loaded the new {{fair-scheduler.xml}}. We need call > {{initScheduler}} before start active services or bring {{refreshAll()}} in > front of {{rm.transitionToActive()}}. *It seems it is also important for > other scheduler*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[ https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406995#comment-15406995 ] Hadoop QA commented on YARN-5333: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 55s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 10s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 37s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 28s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 52m 54s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821971/YARN-5333.08.patch | | JIRA Issue | YARN-5333 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux bb0a82d2a536 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a1f6564 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/12642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit test logs | https://builds.apache.org/job/PreCommit-YARN-Build/12642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12642/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12642/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[ https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406980#comment-15406980 ] Rohith Sharma K S commented on YARN-5333: - The approach seems looks good. Few things to consider # {{refreshXXXWithoutCheck}} does not looks meaning full method name. I think common general pattern can be followed like below. Thoughts? {code} public RefreshXXXResponse refreshXXX(RefreshXXXRequest request){ // ACL and RM check can be combined together try{ refresh(); // This should includes only loading configuration file and update required field. // Success audit log }catch{ // failure audit log } } private void refresh(){ // load configuration filie // refresh XXX fields. } {code} # One of my major concern after seeing patch is skipping checkACL which used to verify user for every transition-to-active. But now it is skipped. Does it fine? cc:/[~jianhe] # Test failure is related to patch change. I think this test can be removed only since behavior is changed after this patch. > Some recovered apps are put into default queue when RM HA > - > > Key: YARN-5333 > URL: https://issues.apache.org/jira/browse/YARN-5333 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: YARN-5333.01.patch, YARN-5333.02.patch, > YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch, > YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch > > > Enable RM HA and use FairScheduler, > {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false, > {{yarn.scheduler.fair.user-as-default-queue}} is set to false. > Reproduce steps: > 1. Start two RMs. > 2. After RMs are running, change both RM's file > {{etc/hadoop/fair-scheduler.xml}}, then add some queues. > 3. Submit some apps to the new added queues. > 4. Stop the active RM, then the standby RM will transit to active and recover > apps. > However the new active RM will put recovered apps into default queue because > it might have not loaded the new {{fair-scheduler.xml}}. We need call > {{initScheduler}} before start active services or bring {{refreshAll()}} in > front of {{rm.transitionToActive()}}. *It seems it is also important for > other scheduler*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5468) Scheduling of long-running applications
[ https://issues.apache.org/jira/browse/YARN-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated YARN-5468: - Attachment: YARN-5468.prototype.patch > Scheduling of long-running applications > --- > > Key: YARN-5468 > URL: https://issues.apache.org/jira/browse/YARN-5468 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacityscheduler, fairscheduler >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-5468.prototype.patch > > > This JIRA is about the scheduling of applications with long-running tasks. > It will include adding support to the YARN for a richer set of scheduling > constraints (such as affinity, anti-affinity, cardinality and time > constraints), and extending the schedulers to take them into account during > placement of containers to nodes. > We plan to have both an online version that will accommodate such requests as > they arrive, as well as a Long-running Application Planner that will make > more global decisions by considering multiple applications at once. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5468) Scheduling of long-running applications
[ https://issues.apache.org/jira/browse/YARN-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated YARN-5468: - Attachment: (was: YARN-5468.prototype.patch) > Scheduling of long-running applications > --- > > Key: YARN-5468 > URL: https://issues.apache.org/jira/browse/YARN-5468 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacityscheduler, fairscheduler >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-5468.prototype.patch > > > This JIRA is about the scheduling of applications with long-running tasks. > It will include adding support to the YARN for a richer set of scheduling > constraints (such as affinity, anti-affinity, cardinality and time > constraints), and extending the schedulers to take them into account during > placement of containers to nodes. > We plan to have both an online version that will accommodate such requests as > they arrive, as well as a Long-running Application Planner that will make > more global decisions by considering multiple applications at once. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5333) Some recovered apps are put into default queue when RM HA
[ https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Gong updated YARN-5333: --- Attachment: YARN-5333.08.patch Fix test case error... > Some recovered apps are put into default queue when RM HA > - > > Key: YARN-5333 > URL: https://issues.apache.org/jira/browse/YARN-5333 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: YARN-5333.01.patch, YARN-5333.02.patch, > YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch, > YARN-5333.06.patch, YARN-5333.07.patch, YARN-5333.08.patch > > > Enable RM HA and use FairScheduler, > {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false, > {{yarn.scheduler.fair.user-as-default-queue}} is set to false. > Reproduce steps: > 1. Start two RMs. > 2. After RMs are running, change both RM's file > {{etc/hadoop/fair-scheduler.xml}}, then add some queues. > 3. Submit some apps to the new added queues. > 4. Stop the active RM, then the standby RM will transit to active and recover > apps. > However the new active RM will put recovered apps into default queue because > it might have not loaded the new {{fair-scheduler.xml}}. We need call > {{initScheduler}} before start active services or bring {{refreshAll()}} in > front of {{rm.transitionToActive()}}. *It seems it is also important for > other scheduler*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2
[ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406938#comment-15406938 ] Li Lu commented on YARN-4061: - Let me revive this thread after the branch merge. From some offline discussion I think our plan is to implement a specialized BufferedMutator so that it can retry when the cluster is down. The benefit of this approach is we do not need to repost the data to buffered mutator, so that saves much memory operations when the cluster is down. We can pretty much reuse some retry logic in our codebase today. The challenge for this design is that we're not persisting anything until the data reaches the HBase cluster. That is to say, with this change we can handle the case when the HBase cluster is down, but cannot handle if collectors themselves are down. If the collector fails when it's retrying, we lose the data. To address this problem, we may use a local journal file to store the state in the buffered mutator. Aggregation status is something we need to recover if collectors fail. However, at the very first phase maybe we can restart everything in the aggregation table on restarts? I know this thread is an old one, but please feel free to chime in since we're targeting to add this feature to the Alpha 2 phase of timeline v2. > [Fault tolerance] Fault tolerant writer for timeline v2 > --- > > Key: YARN-4061 > URL: https://issues.apache.org/jira/browse/YARN-4061 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Labels: YARN-5355 > Attachments: FaulttolerantwriterforTimelinev2.pdf > > > We need to build a timeline writer that can be resistant to backend storage > down time and timeline collector failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5472) WIN_MAX_PATH logic is off by one
[ https://issues.apache.org/jira/browse/YARN-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brook Zhou updated YARN-5472: - Description: The following check is incorrect in DefaultContainerExecutor: if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > WIN_MAX_PATH) should be >=, as the max path is defined as "D:\some 256-character path string" on Windows platforms. was: The following check is incorrect in DefaultContainerExecutor: if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > WIN_MAX_PATH) should be >=, as the max path is defined as "D:\some 256-character path string" > WIN_MAX_PATH logic is off by one > > > Key: YARN-5472 > URL: https://issues.apache.org/jira/browse/YARN-5472 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 > Environment: Windows >Reporter: Brook Zhou >Assignee: Brook Zhou >Priority: Minor > > The following check is incorrect in DefaultContainerExecutor: > if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > > WIN_MAX_PATH) > should be >=, as the max path is defined as "D:\some 256-character path > string" on Windows platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5472) WIN_MAX_PATH logic is off by one
[ https://issues.apache.org/jira/browse/YARN-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brook Zhou updated YARN-5472: - Affects Version/s: 2.6.0 > WIN_MAX_PATH logic is off by one > > > Key: YARN-5472 > URL: https://issues.apache.org/jira/browse/YARN-5472 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 > Environment: Windows >Reporter: Brook Zhou >Assignee: Brook Zhou >Priority: Minor > > The following check is incorrect in DefaultContainerExecutor: > if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > > WIN_MAX_PATH) > should be >=, as the max path is defined as "D:\some 256-character path > string" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5450) Enhance logging for Cluster.java around InetSocketAddress
[ https://issues.apache.org/jira/browse/YARN-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406891#comment-15406891 ] Li Lu commented on YARN-5450: - Patch LGTM. [~sarun] could you please verify if this would be fine with your suggested use case? Thanks! > Enhance logging for Cluster.java around InetSocketAddress > - > > Key: YARN-5450 > URL: https://issues.apache.org/jira/browse/YARN-5450 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn >Reporter: sarun singla >Assignee: Vrushali C >Priority: Minor > Labels: YARN > Attachments: YARN-5450.01.patch > > > We need to add more logging for cluster.java class around " > initialize(InetSocketAddress jobTrackAddr, Configuration conf) " method to > give better logging like about the source of the property. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5472) WIN_MAX_PATH logic is off by one
[ https://issues.apache.org/jira/browse/YARN-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brook Zhou updated YARN-5472: - Description: The following check is incorrect in DefaultContainerExecutor: if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > WIN_MAX_PATH) should be >=, as the max path is defined as "D:\some 256-character path string" was: The following check is incorrect: if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > WIN_MAX_PATH) should be >=, as the max path is defined as "D:\some 256-character path string" > WIN_MAX_PATH logic is off by one > > > Key: YARN-5472 > URL: https://issues.apache.org/jira/browse/YARN-5472 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Environment: Windows >Reporter: Brook Zhou >Assignee: Brook Zhou >Priority: Minor > > The following check is incorrect in DefaultContainerExecutor: > if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > > WIN_MAX_PATH) > should be >=, as the max path is defined as "D:\some 256-character path > string" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5472) WIN_MAX_PATH logic is off by one
Brook Zhou created YARN-5472: Summary: WIN_MAX_PATH logic is off by one Key: YARN-5472 URL: https://issues.apache.org/jira/browse/YARN-5472 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Environment: Windows Reporter: Brook Zhou Assignee: Brook Zhou Priority: Minor The following check is incorrect: if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > WIN_MAX_PATH) should be >=, as the max path is defined as "D:\some 256-character path string" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4091) Add REST API to retrieve scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Ge updated YARN-4091: -- Attachment: YARN-4091.8.patch rebased to trunk > Add REST API to retrieve scheduler activity > --- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > SchedulerActivityManager-TestReport v2.pdf, > SchedulerActivityManager-TestReport.pdf, YARN-4091-design-doc-v1.pdf, > YARN-4091.1.patch, YARN-4091.2.patch, YARN-4091.3.patch, YARN-4091.4.patch, > YARN-4091.5.patch, YARN-4091.5.patch, YARN-4091.6.patch, YARN-4091.7.patch, > YARN-4091.8.patch, YARN-4091.preliminary.1.patch, app_activities v2.json, > app_activities.json, node_activities v2.json, node_activities.json > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406877#comment-15406877 ] Hadoop QA commented on YARN-5384: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 0s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 54s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 58s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 55s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 47s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s {color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 13s {color} | {color:red} hadoop-yarn-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 6s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 49s {color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 76m 39s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.logaggregation.TestAggregatedLogFormat | | | hadoop.yarn.client.api.impl.TestYarnClient | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821940/YARN-5384.v1.patch | | JIRA Issue | YARN-5384 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux bb5449a142d6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a1f6564 | | Default Java | 1.8.0_101 |
[jira] [Commented] (YARN-4091) Add REST API to retrieve scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406870#comment-15406870 ] Hadoop QA commented on YARN-4091: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color} | {color:red} YARN-4091 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821942/YARN-4091.7.patch | | JIRA Issue | YARN-4091 | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12640/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Add REST API to retrieve scheduler activity > --- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > SchedulerActivityManager-TestReport v2.pdf, > SchedulerActivityManager-TestReport.pdf, YARN-4091-design-doc-v1.pdf, > YARN-4091.1.patch, YARN-4091.2.patch, YARN-4091.3.patch, YARN-4091.4.patch, > YARN-4091.5.patch, YARN-4091.5.patch, YARN-4091.6.patch, YARN-4091.7.patch, > YARN-4091.preliminary.1.patch, app_activities v2.json, app_activities.json, > node_activities v2.json, node_activities.json > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5307) Federation Application State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406862#comment-15406862 ] Hadoop QA commented on YARN-5307: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 31s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 45s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 55s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 33s {color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 15m 21s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821953/YARN-5307-YARN-2915-v5.patch | | JIRA Issue | YARN-5307 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux ba80aa7ac143 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | YARN-2915 / 22db8fd | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12639/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12639/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Federation Application State Store internal APIs > > > Key: YARN-5307 > URL: https://issues.apache.org/jira/browse/YARN-5307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krish
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[ https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406853#comment-15406853 ] Hadoop QA commented on YARN-1547: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 38s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 19s {color} | {color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 35 unchanged - 0 fixed = 36 total (was 35) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s {color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 13 new + 208 unchanged - 0 fixed = 221 total (was 208) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 35s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s {color} | {color:red} hadoop-yarn-server-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 22s {color} | {color:red} hadoop-yarn-api in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s {color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 11s {color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 40m 0s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12804094/YARN-1547.prototype.v0.patch | | JIRA Issue | YARN-1547 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 02b02dd84b00 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a1f6564 | | Defau
[jira] [Commented] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406843#comment-15406843 ] Hadoop QA commented on YARN-5382: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 13s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 32s {color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s {color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s {color} | {color:green} branch-2.7 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s {color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s {color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s {color} | {color:green} branch-2.7 passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 13s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s {color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s {color} | {color:green} branch-2.7 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 31 new + 683 unchanged - 7 fixed = 714 total (was 690) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 4582 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 1m 45s {color} | {color:red} The patch 104 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 14s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_101. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m 39s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_101. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 38s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 144m 10s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_101 Failed junit tests | hadoop.yarn.server.resourcemanager.TestClientRMT
[jira] [Updated] (YARN-5307) Federation Application State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5307: - Attachment: YARN-5307-YARN-2915-v5.patch Thanks [~leftnoteasy] for your feedback. I have addressed all of them in v5. > Federation Application State Store internal APIs > > > Key: YARN-5307 > URL: https://issues.apache.org/jira/browse/YARN-5307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5307-YARN-2915-v1.patch, > YARN-5307-YARN-2915-v2.patch, YARN-5307-YARN-2915-v3.patch, > YARN-5307-YARN-2915-v4.patch, YARN-5307-YARN-2915-v5.patch > > > The Federation Application State encapsulates the mapping between an > application and it's _home_ sub-cluster, i.e. the sub-cluster to which it is > submitted to by the Router. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5307) Federation Application State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406827#comment-15406827 ] Subru Krishnan commented on YARN-5307: -- Thanks [~vvasudev] for reviewing the patch. I'll take care of it as soon as [~leftnoteasy] signs off. > Federation Application State Store internal APIs > > > Key: YARN-5307 > URL: https://issues.apache.org/jira/browse/YARN-5307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5307-YARN-2915-v1.patch, > YARN-5307-YARN-2915-v2.patch, YARN-5307-YARN-2915-v3.patch, > YARN-5307-YARN-2915-v4.patch > > > The Federation Application State encapsulates the mapping between an > application and it's _home_ sub-cluster, i.e. the sub-cluster to which it is > submitted to by the Router. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5406) In-memory based implementation of the FederationMembershipStateStore
[ https://issues.apache.org/jira/browse/YARN-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406823#comment-15406823 ] Subru Krishnan commented on YARN-5406: -- +1 on the latest patch. Thanks [~ellenfkh] for addressing our comments, I'll commit it shortly. > In-memory based implementation of the FederationMembershipStateStore > > > Key: YARN-5406 > URL: https://issues.apache.org/jira/browse/YARN-5406 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Ellen Hui > Attachments: YARN-5406-YARN-2915.v0.patch, > YARN-5406-YARN-2915.v1.patch > > > YARN-3662 defines the FederationMembershipStateStore API. This JIRA tracks an > in-memory based implementation which is useful for both single-box testing > and for future unit tests that depend on the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5390) Federation Subcluster Resolver
[ https://issues.apache.org/jira/browse/YARN-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406820#comment-15406820 ] Subru Krishnan edited comment on YARN-5390 at 8/3/16 11:31 PM: --- Thanks [~leftnoteasy] for reviewing the patch. Thanks [~ellenfkh] for the patch. It mostly LGTM, just a couple of minor comments: * Can you add a *YARN_FEDERATION_PREFIX* in {{YarnConfiguration}} as we are going to be using it a lot more in subsequent JIRAs. * Please add findbugs exclusion for {{nodes}} and {{nodes-malformed}} test resources. * I feel the checkstyle warnings are not blockers but will good to fix if it's easy to do so. was (Author: subru): Thanks [~leftnoteasy] for reviewing the patch. It mostly LGTM, just a couple of minor comments: * Can you add a *YARN_FEDERATION_PREFIX* in {{YarnConfiguration}} as we are going to be using it a lot more in subsequent JIRAs. * Please add findbugs exclusion for {{nodes}} and {{nodes-malformed}} test resources. * I feel the checkstyle warnings are not blockers but will good to fix if it's easy to do so. > Federation Subcluster Resolver > -- > > Key: YARN-5390 > URL: https://issues.apache.org/jira/browse/YARN-5390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Carlo Curino >Assignee: Ellen Hui > Attachments: YARN-5390-YARN-2915.v0.patch, > YARN-5390-YARN-2915.v1.patch, YARN-5390-YARN-2915.v2.patch > > > This JIRA tracks effort to create a mechanism to resolve nodes/racks resource > names to sub-cluster identifiers. This is needed by the federation policies > in YARN-5323, YARN-5324, YARN-5325 to operate correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5470: Attachment: YARN-5470.2.patch > Differentiate exactly match with regex in yarn log CLI > -- > > Key: YARN-5470 > URL: https://issues.apache.org/jira/browse/YARN-5470 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5470.1.patch, YARN-5470.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5390) Federation Subcluster Resolver
[ https://issues.apache.org/jira/browse/YARN-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406820#comment-15406820 ] Subru Krishnan commented on YARN-5390: -- Thanks [~leftnoteasy] for reviewing the patch. It mostly LGTM, just a couple of minor comments: * Can you add a *YARN_FEDERATION_PREFIX* in {{YarnConfiguration}} as we are going to be using it a lot more in subsequent JIRAs. * Please add findbugs exclusion for {{nodes}} and {{nodes-malformed}} test resources. * I feel the checkstyle warnings are not blockers but will good to fix if it's easy to do so. > Federation Subcluster Resolver > -- > > Key: YARN-5390 > URL: https://issues.apache.org/jira/browse/YARN-5390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Carlo Curino >Assignee: Ellen Hui > Attachments: YARN-5390-YARN-2915.v0.patch, > YARN-5390-YARN-2915.v1.patch, YARN-5390-YARN-2915.v2.patch > > > This JIRA tracks effort to create a mechanism to resolve nodes/racks resource > names to sub-cluster identifiers. This is needed by the federation policies > in YARN-5323, YARN-5324, YARN-5325 to operate correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5450) Enhance logging for Cluster.java around InetSocketAddress
[ https://issues.apache.org/jira/browse/YARN-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406783#comment-15406783 ] Vrushali C commented on YARN-5450: -- Hi [~sarun] Did you get a chance to look at the patch? thanks Vrushali > Enhance logging for Cluster.java around InetSocketAddress > - > > Key: YARN-5450 > URL: https://issues.apache.org/jira/browse/YARN-5450 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn >Reporter: sarun singla >Assignee: Vrushali C >Priority: Minor > Labels: YARN > Attachments: YARN-5450.01.patch > > > We need to add more logging for cluster.java class around " > initialize(InetSocketAddress jobTrackAddr, Configuration conf) " method to > give better logging like about the source of the property. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3664) Federation PolicyStore internal APIs
[ https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406780#comment-15406780 ] Subru Krishnan commented on YARN-3664: -- Thanks [~leftnoteasy] for your feedback. I agree that using enums wherever possible is better than String. But in the case of types, they are extendible/pluggable and so we see the number growing. For e.g.: We already have 4-5 as part of YARN-5324/YARN-5325 and ideas for a couple more based on how these work out in testing. > Federation PolicyStore internal APIs > > > Key: YARN-3664 > URL: https://issues.apache.org/jira/browse/YARN-3664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3664-YARN-2915-v0.patch, > YARN-3664-YARN-2915-v1.patch, YARN-3664-YARN-2915-v2.patch, > YARN-3664-YARN-2915-v3.patch > > > The federation Policy Store contains information about the capacity > allocations made by users, their mapping to sub-clusters and the policies > that each of the components (Router, AMRMPRoxy, RMs) should enforce -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3664) Federation PolicyStore internal APIs
[ https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406772#comment-15406772 ] Subru Krishnan commented on YARN-3664: -- Thanks [~vvasudev] for reviewing the patch. Yes, I intentionally renamed _addReservationResourcesToProto_ to _addSubClusterInfosToProto_ as it was originally a copy-paste typo. I'll take care of it as soon as [~leftnoteasy] signs off. > Federation PolicyStore internal APIs > > > Key: YARN-3664 > URL: https://issues.apache.org/jira/browse/YARN-3664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3664-YARN-2915-v0.patch, > YARN-3664-YARN-2915-v1.patch, YARN-3664-YARN-2915-v2.patch, > YARN-3664-YARN-2915-v3.patch > > > The federation Policy Store contains information about the capacity > allocations made by users, their mapping to sub-clusters and the policies > that each of the components (Router, AMRMPRoxy, RMs) should enforce -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5471) Adding null pointer checks in ResourceRequest#newInstance
Giovanni Matteo Fumarola created YARN-5471: -- Summary: Adding null pointer checks in ResourceRequest#newInstance Key: YARN-5471 URL: https://issues.apache.org/jira/browse/YARN-5471 Project: Hadoop YARN Issue Type: Bug Components: applications, resourcemanager Reporter: Giovanni Matteo Fumarola Assignee: Giovanni Matteo Fumarola ResourceRequest#newInstance has Priority, Resource and Strings fields. The application master can set these value to null. The proposal is to add null pointer checks in the class. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4091) Add REST API to retrieve scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406757#comment-15406757 ] Hadoop QA commented on YARN-4091: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color} | {color:red} YARN-4091 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821942/YARN-4091.7.patch | | JIRA Issue | YARN-4091 | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12635/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Add REST API to retrieve scheduler activity > --- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > SchedulerActivityManager-TestReport v2.pdf, > SchedulerActivityManager-TestReport.pdf, YARN-4091-design-doc-v1.pdf, > YARN-4091.1.patch, YARN-4091.2.patch, YARN-4091.3.patch, YARN-4091.4.patch, > YARN-4091.5.patch, YARN-4091.5.patch, YARN-4091.6.patch, YARN-4091.7.patch, > YARN-4091.preliminary.1.patch, app_activities v2.json, app_activities.json, > node_activities v2.json, node_activities.json > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4888) Changes in RM container allocation for identifying resource-requests explicitly
[ https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406742#comment-15406742 ] Wangda Tan commented on YARN-4888: -- [~subru], I would still suggest to merge the two tests even if ParameterizedSchedulerTestBase doesn't support parameterized tests. We're trying to merge common parts of schedulers, it's better to avoid creating more duplications. For example, putting allocation-id related tests to separate test classes (like Test"CS/FS"AllocationId), extends from same test class. Beyond that, patch looks good to me. > Changes in RM container allocation for identifying resource-requests > explicitly > --- > > Key: YARN-4888 > URL: https://issues.apache.org/jira/browse/YARN-4888 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch, > YARN-4888-v2.patch, YARN-4888-v3.patch, YARN-4888-v4.patch, > YARN-4888-v5.patch, YARN-4888-v6.patch, YARN-4888.001.patch > > > YARN-4879 puts forward the notion of identifying allocate requests > explicitly. This JIRA is to track the changes in RM app scheduling data > structures to accomplish it. Please refer to the design doc in the parent > JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4091) Add REST API to retrieve scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Ge updated YARN-4091: -- Attachment: YARN-4091.7.patch > Add REST API to retrieve scheduler activity > --- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > SchedulerActivityManager-TestReport v2.pdf, > SchedulerActivityManager-TestReport.pdf, YARN-4091-design-doc-v1.pdf, > YARN-4091.1.patch, YARN-4091.2.patch, YARN-4091.3.patch, YARN-4091.4.patch, > YARN-4091.5.patch, YARN-4091.5.patch, YARN-4091.6.patch, YARN-4091.7.patch, > YARN-4091.preliminary.1.patch, app_activities v2.json, app_activities.json, > node_activities v2.json, node_activities.json > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-3664) Federation PolicyStore internal APIs
[ https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406711#comment-15406711 ] Wangda Tan edited comment on YARN-3664 at 8/3/16 10:14 PM: --- [~subru], One minor comment: Can we change getType of SubClusterPolicyConfiguration from String to enum. Using enum should be much better for readability if the type comes from limited choices. And this could also avoid potential bugs. was (Author: leftnoteasy): [~subru], One minor comment: Can we change getType of > Federation PolicyStore internal APIs > > > Key: YARN-3664 > URL: https://issues.apache.org/jira/browse/YARN-3664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3664-YARN-2915-v0.patch, > YARN-3664-YARN-2915-v1.patch, YARN-3664-YARN-2915-v2.patch, > YARN-3664-YARN-2915-v3.patch > > > The federation Policy Store contains information about the capacity > allocations made by users, their mapping to sub-clusters and the policies > that each of the components (Router, AMRMPRoxy, RMs) should enforce -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v1.patch YARN-5384.v1.patch adds priority to the ReservationSystem submission API by adding another field to the ReservationDefinition class. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3664) Federation PolicyStore internal APIs
[ https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406711#comment-15406711 ] Wangda Tan commented on YARN-3664: -- [~subru], One minor comment: Can we change getType of > Federation PolicyStore internal APIs > > > Key: YARN-3664 > URL: https://issues.apache.org/jira/browse/YARN-3664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3664-YARN-2915-v0.patch, > YARN-3664-YARN-2915-v1.patch, YARN-3664-YARN-2915-v2.patch, > YARN-3664-YARN-2915-v3.patch > > > The federation Policy Store contains information about the capacity > allocations made by users, their mapping to sub-clusters and the policies > that each of the components (Router, AMRMPRoxy, RMs) should enforce -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406706#comment-15406706 ] Hadoop QA commented on YARN-5470: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 38s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 10 new + 78 unchanged - 5 fixed = 88 total (was 83) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 21s {color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 20m 4s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821935/YARN-5470.1.patch | | JIRA Issue | YARN-5470 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 9310e8390fe3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a1f6564 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/12634/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/12634/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt | | unit test logs | https://builds.apache.org/job/PreCommit-YARN-Build/12634/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12634/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12634/console | | Powered by | Apache Ye
[jira] [Commented] (YARN-5307) Federation Application State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406701#comment-15406701 ] Wangda Tan commented on YARN-5307: -- Hi [~subru], Few comments: - For the responsibility of FederationApplicationStateStore, what else state you plan to add for applications? What I can see is it only manages home-sub-cluster of applications, renaming it to FederationApplicationHomeSubClusterStore if no more items are planned? - For ApplicationHomeSubClusterMap and all related response / request / method name, remove "map" from these names? I was trying to find map-like data structure in definition.. - Cannot figure out who will use FederationStateStore and how it will be used, should we move it to more related JIRA? > Federation Application State Store internal APIs > > > Key: YARN-5307 > URL: https://issues.apache.org/jira/browse/YARN-5307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5307-YARN-2915-v1.patch, > YARN-5307-YARN-2915-v2.patch, YARN-5307-YARN-2915-v3.patch, > YARN-5307-YARN-2915-v4.patch > > > The Federation Application State encapsulates the mapping between an > application and it's _home_ sub-cluster, i.e. the sub-cluster to which it is > submitted to by the Router. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5390) Federation Subcluster Resolver
[ https://issues.apache.org/jira/browse/YARN-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406683#comment-15406683 ] Wangda Tan commented on YARN-5390: -- Thanks [~ellenfkh], I don't have strong opinions about using non-static or static, it totally depends on how it will be used. If lots of modules will use it, static class could be a better choice, if only limited modules will use it, and they're all related to federation, keep it non-static and can be get from FederationContext will be also good. For general purpose of this JIRA, the patch looks fine. I hope someone who is more familiar with federation code could take a deeper look at this patch. cc: [~subru] / [~curino]. > Federation Subcluster Resolver > -- > > Key: YARN-5390 > URL: https://issues.apache.org/jira/browse/YARN-5390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Carlo Curino >Assignee: Ellen Hui > Attachments: YARN-5390-YARN-2915.v0.patch, > YARN-5390-YARN-2915.v1.patch, YARN-5390-YARN-2915.v2.patch > > > This JIRA tracks effort to create a mechanism to resolve nodes/racks resource > names to sub-cluster identifiers. This is needed by the federation policies > in YARN-5323, YARN-5324, YARN-5325 to operate correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5470: Attachment: YARN-5470.1.patch > Differentiate exactly match with regex in yarn log CLI > -- > > Key: YARN-5470 > URL: https://issues.apache.org/jira/browse/YARN-5470 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5470.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
Xuan Gong created YARN-5470: --- Summary: Differentiate exactly match with regex in yarn log CLI Key: YARN-5470 URL: https://issues.apache.org/jira/browse/YARN-5470 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-5382: - Attachment: YARN-5382-branch-2.7.12.patch YARN-5382.12.patch Uploading v12 that addresses javadoc warnings. > RM does not audit log kill request for active applications > -- > > Key: YARN-5382 > URL: https://issues.apache.org/jira/browse/YARN-5382 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Vrushali C > Attachments: YARN-5382-branch-2.7.01.patch, > YARN-5382-branch-2.7.02.patch, YARN-5382-branch-2.7.03.patch, > YARN-5382-branch-2.7.04.patch, YARN-5382-branch-2.7.05.patch, > YARN-5382-branch-2.7.09.patch, YARN-5382-branch-2.7.10.patch, > YARN-5382-branch-2.7.11.patch, YARN-5382-branch-2.7.12.patch, > YARN-5382.06.patch, YARN-5382.07.patch, YARN-5382.08.patch, > YARN-5382.09.patch, YARN-5382.10.patch, YARN-5382.11.patch, YARN-5382.12.patch > > > ClientRMService will audit a kill request but only if it either fails to > issue the kill or if the kill is sent to an already finished application. It > does not create a log entry when the application is active which is arguably > the most important case to audit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5327) API changes required to support recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405299#comment-15405299 ] Sangeetha Abdu Jyothi edited comment on YARN-5327 at 8/3/16 9:16 PM: - Please note that the failed test is unrelated to this patch. was (Author: ajsangeetha): Please note that the failed test in unrelated to this patch. > API changes required to support recurring reservations in the YARN > ReservationSystem > > > Key: YARN-5327 > URL: https://issues.apache.org/jira/browse/YARN-5327 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Attachments: YARN-5327.001.patch, YARN-5327.002.patch, > YARN-5327.003.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to track the changes needed > in ApplicationClientProtocol to accomplish it. Please refer to the design doc > in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5448) Resource in Cluster Metrics is not sum of resources in all nodes of all partitions
[ https://issues.apache.org/jira/browse/YARN-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406613#comment-15406613 ] Wangda Tan commented on YARN-5448: -- bq. One alternative i can think of is show these columns only when partitions are not mapped to queues. and if value is zero then dont show, thoughts ? Sounds like a plan bq. One view point what i had for this was captured in the above comment , but well again its a view point and debatable so dont have any hard restrictions on having it. This is majorly for answering questions like "why my total cluster resource cannot be utilized", it should be already very clear for resource under each partition even without this patch. bq. May be i did not get the rationale behind "total non-usable resources" would be better than "non-usable nodes", can elaborate more on your view on this ? Because resources of nodes should be heterogeneous. In our existing YARN web UI, it shows total resource usage instead of total nodes usage (how many nodes are using by applications). So total non-usable resources will be easier for user to do calculations because non-usable = total - used - available > Resource in Cluster Metrics is not sum of resources in all nodes of all > partitions > -- > > Key: YARN-5448 > URL: https://issues.apache.org/jira/browse/YARN-5448 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager, webapp >Affects Versions: 2.7.2 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: NodesPage.png, schedulerPage.png > > > Currently Resource info from Cluster Metrics are got from Queue Metrics's > *available resource + allocated resource*. Hence if there are some nodes > which belongs to partition but if that partition is not associated with any > queue then in the capacity scheduler partition hierarchy shows this nodes > resources under its partition but Cluster metrics doesn't show. > Apart from this in the Nodes page too Metrics overview table is shown. So if > we show Resource info from Queue Metrics User will not be able to co relate > it. (have attached the images for the same) > IIUC idea of not showing in the *Metrics overview table* is to highlight that > configuration is not proper. This needs to be some how conveyed through > parititon-by-queue-hierarchy chart. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406600#comment-15406600 ] Vrushali C commented on YARN-5382: -- Will quickly fix the javadoc warnings and one of the findbugs issue. I am not sure I can fix the other findbugs (the one about 9 params instead of 7, there are already 8 params in that function). > RM does not audit log kill request for active applications > -- > > Key: YARN-5382 > URL: https://issues.apache.org/jira/browse/YARN-5382 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Vrushali C > Attachments: YARN-5382-branch-2.7.01.patch, > YARN-5382-branch-2.7.02.patch, YARN-5382-branch-2.7.03.patch, > YARN-5382-branch-2.7.04.patch, YARN-5382-branch-2.7.05.patch, > YARN-5382-branch-2.7.09.patch, YARN-5382-branch-2.7.10.patch, > YARN-5382-branch-2.7.11.patch, YARN-5382.06.patch, YARN-5382.07.patch, > YARN-5382.08.patch, YARN-5382.09.patch, YARN-5382.10.patch, YARN-5382.11.patch > > > ClientRMService will audit a kill request but only if it either fails to > issue the kill or if the kill is sent to an already finished application. It > does not create a log entry when the application is active which is arguably > the most important case to audit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406595#comment-15406595 ] Hadoop QA commented on YARN-5382: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 42s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 56s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 214 unchanged - 1 fixed = 216 total (was 215) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 19s {color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 4 new + 963 unchanged - 0 fixed = 967 total (was 963) {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m 2s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 53m 41s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821911/YARN-5382.11.patch | | JIRA Issue | YARN-5382 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 9df726fe0192 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / db4a61d | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/12632/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/12632/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12632/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://
[jira] [Commented] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[ https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406555#comment-15406555 ] Hudson commented on YARN-5469: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10207 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10207/]) YARN-5469. Increase timeout of TestAmFilter.testFilter. Contributed by (jlowe: rev db4a61dc617954cf448a0d44cb4ac79f2abf9db3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/amfilter/TestAmFilter.java > Increase timeout of TestAmFilter.testFilter > --- > > Key: YARN-5469 > URL: https://issues.apache.org/jira/browse/YARN-5469 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Minor > Fix For: 2.8.0, 2.7.4 > > Attachments: YARN-5469.001.patch > > > Timeout is currently only 1 second. Saw a timeout failure > {noformat} > java.lang.Exception: test timed out after 1000 milliseconds > at java.util.zip.ZipFile.getEntry(Native Method) > at java.util.zip.ZipFile.getEntry(ZipFile.java:311) > at java.util.jar.JarFile.getEntry(JarFile.java:240) > at java.util.jar.JarFile.getJarEntry(JarFile.java:223) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841) > at sun.misc.URLClassPath.getResource(URLClassPath.java:199) > at java.net.URLClassLoader$1.run(URLClassLoader.java:364) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
[jira] [Resolved] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[ https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe resolved YARN-5469. -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.7.4 2.8.0 Thanks, Eric! I committed this to trunk, branch-2, branch-2.8, and branch-2.7. > Increase timeout of TestAmFilter.testFilter > --- > > Key: YARN-5469 > URL: https://issues.apache.org/jira/browse/YARN-5469 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Minor > Fix For: 2.8.0, 2.7.4 > > Attachments: YARN-5469.001.patch > > > Timeout is currently only 1 second. Saw a timeout failure > {noformat} > java.lang.Exception: test timed out after 1000 milliseconds > at java.util.zip.ZipFile.getEntry(Native Method) > at java.util.zip.ZipFile.getEntry(ZipFile.java:311) > at java.util.jar.JarFile.getEntry(JarFile.java:240) > at java.util.jar.JarFile.getJarEntry(JarFile.java:223) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841) > at sun.misc.URLClassPath.getResource(URLClassPath.java:199) > at java.net.URLClassLoader$1.run(URLClassLoader.java:364) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
[jira] [Updated] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-5382: - Attachment: YARN-5382.11.patch YARN-5382-branch-2.7.11.patch Thanks [~jianhe], appreciate the feedback! Uploading v11 with suggested changes. Also, the "greater then" wording in the patch on trunk came in because the existing 2.7 codebase has those. But I have fixed that for both trunk and 2.7 in the v11 patch. > RM does not audit log kill request for active applications > -- > > Key: YARN-5382 > URL: https://issues.apache.org/jira/browse/YARN-5382 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Vrushali C > Attachments: YARN-5382-branch-2.7.01.patch, > YARN-5382-branch-2.7.02.patch, YARN-5382-branch-2.7.03.patch, > YARN-5382-branch-2.7.04.patch, YARN-5382-branch-2.7.05.patch, > YARN-5382-branch-2.7.09.patch, YARN-5382-branch-2.7.10.patch, > YARN-5382-branch-2.7.11.patch, YARN-5382.06.patch, YARN-5382.07.patch, > YARN-5382.08.patch, YARN-5382.09.patch, YARN-5382.10.patch, YARN-5382.11.patch > > > ClientRMService will audit a kill request but only if it either fails to > issue the kill or if the kill is sent to an already finished application. It > does not create a log entry when the application is active which is arguably > the most important case to audit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[ https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406512#comment-15406512 ] Jason Lowe commented on YARN-5469: -- +1 lgtm. One second test timeouts are clearly too low. A JVM or I/O hiccup while loading classes can blow through that as shown by the stack traceback. Committing this. > Increase timeout of TestAmFilter.testFilter > --- > > Key: YARN-5469 > URL: https://issues.apache.org/jira/browse/YARN-5469 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Minor > Attachments: YARN-5469.001.patch > > > Timeout is currently only 1 second. Saw a timeout failure > {noformat} > java.lang.Exception: test timed out after 1000 milliseconds > at java.util.zip.ZipFile.getEntry(Native Method) > at java.util.zip.ZipFile.getEntry(ZipFile.java:311) > at java.util.jar.JarFile.getEntry(JarFile.java:240) > at java.util.jar.JarFile.getJarEntry(JarFile.java:223) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841) > at sun.misc.URLClassPath.getResource(URLClassPath.java:199) > at java.net.URLClassLoader$1.run(URLClassLoader.java:364) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.ja
[jira] [Commented] (YARN-5406) In-memory based implementation of the FederationMembershipStateStore
[ https://issues.apache.org/jira/browse/YARN-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406502#comment-15406502 ] Hadoop QA commented on YARN-5406: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 51s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 47s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s {color} | {color:green} YARN-2915 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s {color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 16m 47s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821899/YARN-5406-YARN-2915.v1.patch | | JIRA Issue | YARN-5406 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 7dccb0e8b201 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | YARN-2915 / 22db8fd | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12631/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12631/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > In-memory based implementation of the FederationMembershipStateStore > > > Key: YARN-5406 > URL: https://issues.apache.org/jira/browse/YARN-5406 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Ellen Hui > Attachments: YARN-5406-YARN-2915.v0.patch, >
[jira] [Updated] (YARN-4717) TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IllegalArgumentException from cleanup
[ https://issues.apache.org/jira/browse/YARN-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-4717: - Fix Version/s: (was: 2.9.0) 2.7.4 2.8.0 Thanks, [~templedf]! I committed this to branch-2.8 and branch-2.7 as well. > TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails > Intermittently due to IllegalArgumentException from cleanup > --- > > Key: YARN-4717 > URL: https://issues.apache.org/jira/browse/YARN-4717 > Project: Hadoop YARN > Issue Type: Test > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Fix For: 2.8.0, 2.7.4 > > Attachments: YARN-4717.001.patch > > > The same issue that was resolved by [~zxu] in YARN-3602 is back. Looks like > the commons-io package throws an IAE instead of an IOE now if the directory > doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5462) TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently
[ https://issues.apache.org/jira/browse/YARN-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406476#comment-15406476 ] Hudson commented on YARN-5462: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10205 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10205/]) YARN-5462. TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown (jlowe: rev db646540f094077941b56ed681a4f3e5853f5b7f) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java > TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails > intermittently > -- > > Key: YARN-5462 > URL: https://issues.apache.org/jira/browse/YARN-5462 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 2.8.0, 2.6.5, 2.7.4 > > Attachments: YARN-5462.001.patch > > > {noformat} > java.io.IOException: Failed on local exception: java.io.IOException: > Connection reset by peer; Host Details : local host is: > "slave-02.adcd.infra.corp.gq1.yahoo.com/69.147.96.229"; destination host is: > "127.0.0.1":12345; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776) > at org.apache.hadoop.ipc.Client.call(Client.java:1457) > at org.apache.hadoop.ipc.Client.call(Client.java:1390) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy78.startContainers(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:101) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown.startContainer(TestNodeManagerShutdown.java:248) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown(TestNodeStatusUpdater.java:1492) > Caused by: java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > at > org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) > at java.io.FilterInputStream.read(FilterInputStream.java:133) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read(BufferedInputStream.java:265) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:508) > at java.io.DataInputStream.readInt(DataInputStream.java:387) > at > org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1730) > at > org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1078) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:977) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
[ https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406477#comment-15406477 ] Hudson commented on YARN-4280: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10205 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10205/]) YARN-4280. CapacityScheduler reservations may not prevent indefinite (jlowe: rev 4d92aefd35d4517d9435d81bafdec0d77905a7a0) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/IncreaseContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AbstractContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java > CapacityScheduler reservations may not prevent indefinite postponement on a > busy cluster > > > Key: YARN-4280 > URL: https://issues.apache.org/jira/browse/YARN-4280 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.6.1, 2.8.0, 2.7.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 2.8.0 > > Attachments: YARN-4280-branch-2.009.patch, > YARN-4280-branch-2.8.001.patch, YARN-4280-branch-2.8.002.patch, > YARN-4280-branch-2.8.003.patch, YARN-4280-branch-2.8.004.patch, > YARN-4280-branch-2.8.005.patch, YARN-4280.001.patch, YARN-4280.002.patch, > YARN-4280.003.patch, YARN-4280.004.patch, YARN-4280.005.patch, > YARN-4280.006.patch, YARN-4280.007.patch, YARN-4280.008.patch, > YARN-4280.008_.patch, YARN-4280.009.patch, YARN-4280.010.patch, > YARN-4280.011.patch, YARN-4280.012.patch, YARN-4280.013.patch, > YARN-4280.014.patch > > > Consider the following scenario: > There are 2 queues A(25% of the total capacity) and B(75%), both can run at > total cluster capacity. There are 2 applications, appX that runs on Queue A, > always asking for 1G containers(non-AM) and appY runs on Queue B asking for 2 > GB containers. > The user limit is high enough for the application to reach 100% of the > cluster resource. > appX is running at total cluster capacity, full with 1G containers releasing > only one container at a time. appY comes in with a request of 2GB container > but only 1 GB is free. Ideally, since appY is in the underserved queue, it > has higher priority and should reserve for its 2 GB request. Since this > request puts the alloc+reserve above total capacity of the cluster, > reservation is not made. appX comes in with a 1GB request and since 1GB is > still available, the request is allocated. > This can continue indefinitely causing priority inversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
[ https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406467#comment-15406467 ] Kuhu Shukla commented on YARN-4280: --- Thank you all for the detailed reviews. > CapacityScheduler reservations may not prevent indefinite postponement on a > busy cluster > > > Key: YARN-4280 > URL: https://issues.apache.org/jira/browse/YARN-4280 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.6.1, 2.8.0, 2.7.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Fix For: 2.8.0 > > Attachments: YARN-4280-branch-2.009.patch, > YARN-4280-branch-2.8.001.patch, YARN-4280-branch-2.8.002.patch, > YARN-4280-branch-2.8.003.patch, YARN-4280-branch-2.8.004.patch, > YARN-4280-branch-2.8.005.patch, YARN-4280.001.patch, YARN-4280.002.patch, > YARN-4280.003.patch, YARN-4280.004.patch, YARN-4280.005.patch, > YARN-4280.006.patch, YARN-4280.007.patch, YARN-4280.008.patch, > YARN-4280.008_.patch, YARN-4280.009.patch, YARN-4280.010.patch, > YARN-4280.011.patch, YARN-4280.012.patch, YARN-4280.013.patch, > YARN-4280.014.patch > > > Consider the following scenario: > There are 2 queues A(25% of the total capacity) and B(75%), both can run at > total cluster capacity. There are 2 applications, appX that runs on Queue A, > always asking for 1G containers(non-AM) and appY runs on Queue B asking for 2 > GB containers. > The user limit is high enough for the application to reach 100% of the > cluster resource. > appX is running at total cluster capacity, full with 1G containers releasing > only one container at a time. appY comes in with a request of 2GB container > but only 1 GB is free. Ideally, since appY is in the underserved queue, it > has higher priority and should reserve for its 2 GB request. Since this > request puts the alloc+reserve above total capacity of the cluster, > reservation is not made. appX comes in with a 1GB request and since 1GB is > still available, the request is allocated. > This can continue indefinitely causing priority inversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5406) In-memory based implementation of the FederationMembershipStateStore
[ https://issues.apache.org/jira/browse/YARN-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ellen Hui updated YARN-5406: Attachment: YARN-5406-YARN-2915.v1.patch Addressed feedback from [~subru] and [~jianhe]. > In-memory based implementation of the FederationMembershipStateStore > > > Key: YARN-5406 > URL: https://issues.apache.org/jira/browse/YARN-5406 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Ellen Hui > Attachments: YARN-5406-YARN-2915.v0.patch, > YARN-5406-YARN-2915.v1.patch > > > YARN-3662 defines the FederationMembershipStateStore API. This JIRA tracks an > in-memory based implementation which is useful for both single-box testing > and for future unit tests that depend on the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5462) TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently
[ https://issues.apache.org/jira/browse/YARN-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406441#comment-15406441 ] Jason Lowe commented on YARN-5462: -- +1 lgtm. Committing this. > TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails > intermittently > -- > > Key: YARN-5462 > URL: https://issues.apache.org/jira/browse/YARN-5462 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: YARN-5462.001.patch > > > {noformat} > java.io.IOException: Failed on local exception: java.io.IOException: > Connection reset by peer; Host Details : local host is: > "slave-02.adcd.infra.corp.gq1.yahoo.com/69.147.96.229"; destination host is: > "127.0.0.1":12345; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776) > at org.apache.hadoop.ipc.Client.call(Client.java:1457) > at org.apache.hadoop.ipc.Client.call(Client.java:1390) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy78.startContainers(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:101) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown.startContainer(TestNodeManagerShutdown.java:248) > at > org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown(TestNodeStatusUpdater.java:1492) > Caused by: java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > at > org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) > at java.io.FilterInputStream.read(FilterInputStream.java:133) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read(BufferedInputStream.java:265) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:508) > at java.io.DataInputStream.readInt(DataInputStream.java:387) > at > org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1730) > at > org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1078) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:977) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4232) TopCLI console support for HA mode
[ https://issues.apache.org/jira/browse/YARN-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406440#comment-15406440 ] Hadoop QA commented on YARN-4232: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 46s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 29s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 5 new + 152 unchanged - 0 fixed = 157 total (was 152) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 26s {color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 20m 24s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821889/YARN-4232.003.patch | | JIRA Issue | YARN-4232 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux b71d2945590e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 580a833 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/12630/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/12630/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt | | unit test logs | https://builds.apache.org/job/PreCommit-YARN-Build/12630/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12630/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12630/console | | Powered by | Apac
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406424#comment-15406424 ] Naganarasimha G R commented on YARN-5287: - I verified the latest patch and seems to rectify all the comments, so hope i did not miss any nits in the patch, [~vvasudev] & [~rohithsharma] anything else needs to be addressed? > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch, YARN-5287.005.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4624) NPE in PartitionQueueCapacitiesInfo while accessing Schduler UI
[ https://issues.apache.org/jira/browse/YARN-4624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406418#comment-15406418 ] Naganarasimha G R commented on YARN-4624: - Thanks for the patch [~sunilg], Applied patch compiled and tested in an installation, seems like everything is working on top of trunk code. If no one has any concerns will commit this patch tomorrow ! > NPE in PartitionQueueCapacitiesInfo while accessing Schduler UI > --- > > Key: YARN-4624 > URL: https://issues.apache.org/jira/browse/YARN-4624 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: SchedulerUIWithOutLabelMapping.png, YARN-2674-002.patch, > YARN-4624-003.patch, YARN-4624.4.patch, YARN-4624.patch > > > Scenario: > === > Configure nodelables and add to cluster > Start the cluster > {noformat} > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.PartitionQueueCapacitiesInfo.getMaxAMLimitPercentage(PartitionQueueCapacitiesInfo.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(CapacitySchedulerPage.java:163) > at > org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderLeafQueueInfoWithPartition(CapacitySchedulerPage.java:105) > at > org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.render(CapacitySchedulerPage.java:94) > at > org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69) > at > org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79) > at org.apache.hadoop.yarn.webapp.View.render(View.java:235) > at > org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:43) > at > org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117) > at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$LI._(Hamlet.java:7702) > at > org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueueBlock.render(CapacitySchedulerPage.java:293) > at > org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69) > at > org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79) > at org.apache.hadoop.yarn.webapp.View.render(View.java:235) > at > org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:43) > at > org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117) > at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$LI._(Hamlet.java:7702) > at > org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueuesBlock.render(CapacitySchedulerPage.java:447) > at > org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69) > at > org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79) > at org.apache.hadoop.yarn.webapp.View.render(View.java:235) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5334) [YARN-3368] Introduce REFRESH button in various UI pages
[ https://issues.apache.org/jira/browse/YARN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406413#comment-15406413 ] Wangda Tan commented on YARN-5334: -- Thanks so much for working on this! [~Sreenath], this is extremely helpful for end user experiences. I did some tests for this, so far so good. [~sunilg], I'm not quite sure about what 2.-4. means in your comments, I didn't see issues here. > [YARN-3368] Introduce REFRESH button in various UI pages > > > Key: YARN-5334 > URL: https://issues.apache.org/jira/browse/YARN-5334 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Sunil G >Assignee: Sreenath Somarajapuram > Attachments: YARN-5334-YARN-3368-0001.patch > > > It will be better if we have a common Refresh button in all pages to get the > latest information in all tables such as apps/nodes/queue etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4280) CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster
[ https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406411#comment-15406411 ] Jason Lowe commented on YARN-4280: -- +1 for the branch-2.8 patch as well. Committing this. > CapacityScheduler reservations may not prevent indefinite postponement on a > busy cluster > > > Key: YARN-4280 > URL: https://issues.apache.org/jira/browse/YARN-4280 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.6.1, 2.8.0, 2.7.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: YARN-4280-branch-2.009.patch, > YARN-4280-branch-2.8.001.patch, YARN-4280-branch-2.8.002.patch, > YARN-4280-branch-2.8.003.patch, YARN-4280-branch-2.8.004.patch, > YARN-4280-branch-2.8.005.patch, YARN-4280.001.patch, YARN-4280.002.patch, > YARN-4280.003.patch, YARN-4280.004.patch, YARN-4280.005.patch, > YARN-4280.006.patch, YARN-4280.007.patch, YARN-4280.008.patch, > YARN-4280.008_.patch, YARN-4280.009.patch, YARN-4280.010.patch, > YARN-4280.011.patch, YARN-4280.012.patch, YARN-4280.013.patch, > YARN-4280.014.patch > > > Consider the following scenario: > There are 2 queues A(25% of the total capacity) and B(75%), both can run at > total cluster capacity. There are 2 applications, appX that runs on Queue A, > always asking for 1G containers(non-AM) and appY runs on Queue B asking for 2 > GB containers. > The user limit is high enough for the application to reach 100% of the > cluster resource. > appX is running at total cluster capacity, full with 1G containers releasing > only one container at a time. appY comes in with a request of 2GB container > but only 1 GB is free. Ideally, since appY is in the underserved queue, it > has higher priority and should reserve for its 2 GB request. Since this > request puts the alloc+reserve above total capacity of the cluster, > reservation is not made. appX comes in with a 1GB request and since 1GB is > still available, the request is allocated. > This can continue indefinitely causing priority inversion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4232) TopCLI console support for HA mode
[ https://issues.apache.org/jira/browse/YARN-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-4232: --- Attachment: YARN-4232.003.patch > TopCLI console support for HA mode > -- > > Key: YARN-4232 > URL: https://issues.apache.org/jira/browse/YARN-4232 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: 0001-YARN-4232.patch, 0002-YARN-4232.patch, > YARN-4232.003.patch > > > *Steps to reproduce* > Start Top command in YARN in HA mode > ./yarn top > {noformat} > usage: yarn top > -cols Number of columns on the terminal > -delay The refresh delay(in seconds), default is 3 seconds > -help Print usage; for help while the tool is running press 'h' > + Enter > -queuesComma separated list of queues to restrict applications > -rows Number of rows on the terminal > -types Comma separated list of types to restrict applications, > case sensitive(though the display is lower case) > -users Comma separated list of users to restrict applications > {noformat} > Execute *for help while the tool is running press 'h' + Enter* while top > tool is running > Exception is thrown in console continuously > {noformat} > 15/10/07 14:59:28 ERROR cli.TopCLI: Could not fetch RM start time > java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204) > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:589) > at java.net.Socket.connect(Socket.java:538) > at sun.net.NetworkClient.doConnect(NetworkClient.java:180) > at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) > at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) > at sun.net.www.http.HttpClient.(HttpClient.java:211) > at sun.net.www.http.HttpClient.New(HttpClient.java:308) > at sun.net.www.http.HttpClient.New(HttpClient.java:326) > at > sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1168) > at > sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1104) > at > sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:998) > at > sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:932) > at > org.apache.hadoop.yarn.client.cli.TopCLI.getRMStartTime(TopCLI.java:742) > at org.apache.hadoop.yarn.client.cli.TopCLI.run(TopCLI.java:467) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.yarn.client.cli.TopCLI.main(TopCLI.java:420) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5451) Container localizers that hang are not cleaned up
[ https://issues.apache.org/jira/browse/YARN-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406362#comment-15406362 ] Jason Lowe commented on YARN-5451: -- Note that we can get localizers to stop today (e.g.: when the corresponding container being localized is killed), but only if the localizer is well-behaved. So even with large sized resources as long as the localizer heartbeat thread is still heartbeating to the NM and can respond properly to heartbeat commands things end up working out OK. It becomes a problem when the localizer _doesn't_ behave properly and needs external intervention. I don't think that's common in practice, but it does happen sometimes. > Container localizers that hang are not cleaned up > - > > Key: YARN-5451 > URL: https://issues.apache.org/jira/browse/YARN-5451 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe > > I ran across an old, rogue process on one of our nodes. It apparently was a > container localizer that somehow entered an infinite loop during startup. > The NM never cleaned up this broken localizer, so it happily ran forever. > The NM needs to do a better job of tracking localizers, including killing > them if they appear to be hung/broken. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5429) Fix @return related javadoc warnings in yarn-api
[ https://issues.apache.org/jira/browse/YARN-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406273#comment-15406273 ] Hadoop QA commented on YARN-5429: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 50s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s {color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api: The patch generated 0 new + 100 unchanged - 1 fixed = 100 total (was 101) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s {color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api generated 0 new + 125 unchanged - 31 fixed = 125 total (was 156) {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s {color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 14m 7s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821866/YARN-5429.04.patch | | JIRA Issue | YARN-5429 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 03b27480ce69 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 58db263 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12629/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12629/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Fix @return related javadoc warnings in yarn-api > > > Key: YARN-5429 > URL: https://issues.apache.org/jira/browse/YARN-5429 > Project: Hadoop YARN > Issue Type: Sub-task >
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406249#comment-15406249 ] Shane Kumpf commented on YARN-5428: --- They most certainly can. The user would set the docker client config directory at job submission time to point to where their personal config.json file lives. This is simply setting a client config directory. How you and your users decide to use it/enforce it in your large environment is up to you. This does not force you into storing your credentials in the config. The client config is also used for other client config as well, such as HTTP proxy settings, which could be quite useful to define cluster wide. There is no default setting enforced by YARN. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch, YARN-5428.004.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5429) Fix @return related javadoc warnings in yarn-api
[ https://issues.apache.org/jira/browse/YARN-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-5429: - Attachment: YARN-5429.04.patch Thanks [~varun_saxena], uploading v4 that addresses the javadoc issue related to this patch. {code} [machine-channapattan hadoop (trunk)]$ diff ../YARN-5429.04.patch ../YARN-5429.03.patch 22c22 < index 0b65e5c..e24ebdf 100644 --- > index 0b65e5c..462f6a2 100644 29c29 < + * @return the list of {@link ContainerResourceChangeRequest} --- > + * @return the list of {@link ContainerResourceChangeRequest}> [machine-13-channapattan hadoop (trunk)]$ {code} > Fix @return related javadoc warnings in yarn-api > > > Key: YARN-5429 > URL: https://issues.apache.org/jira/browse/YARN-5429 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-5429.01.patch, YARN-5429.02.patch, > YARN-5429.03.patch, YARN-5429.04.patch > > > As part of YARN-4977, filing a subtask to fix a subset of the javadoc > warnings in yarn-api. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406226#comment-15406226 ] Allen Wittenauer commented on YARN-5428: ... and none of that answers the key question: Why is the admin providing the passwords and not the user? Why can't a user provide a config.json as part of their job submission? All users having the exact same access to every docker registry seems like a huge fail in any large (read: enterprise) environment. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch, YARN-5428.004.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5453) FairScheduler#update may skip update demand resource of child queue/app if current demand reached maxResource
[ https://issues.apache.org/jira/browse/YARN-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406221#comment-15406221 ] Hadoop QA commented on YARN-5453: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 54s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 7s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 53m 11s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation | | | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821858/YARN-5453.01.patch | | JIRA Issue | YARN-5453 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux e57a2da6689e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 2d82276 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/12628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit test logs | https://builds.apache.org/job/PreCommit-YARN-Build/12628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12628/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn
[jira] [Commented] (YARN-5455) LinuxContainerExecutor needs Javadocs
[ https://issues.apache.org/jira/browse/YARN-5455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406194#comment-15406194 ] Varun Vasudev commented on YARN-5455: - Thanks for the patch [~templedf]! {code} + * + * + * {@code YARN_CONTAINER_RUNTIME_DOCKER_IMAGE} names which image + * will be used to launch the Docker container. + * + * + * {@code YARN_CONTAINER_RUNTIME_DOCKER_IMAGE_FILE} is currently ignored. + * + * + * {@code YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE} controls + * whether the Docker container's default command is overridden. When set + * to {@code true}, the Docker container's command will be + * {@code bash }. When unset or set to {@code false} + * the Docker container's default command is used. + * + * + * {@code YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK} sets the + * network type to be used by the Docker container. It must be a valid + * value as determined by the + * {@code yarn.nodemanager.runtime.linux.docker.allowed-container-networks} + * property. + * + * + * {@code YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER} + * controls whether the Docker container is a privileged container. In order + * to use privileged containers, the + * {@code yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed} + * property must be set to {@code true}, and the application owner must + * appear in the value of the + * {@code yarn.nodemanager.runtime.linux.docker.privileged-containers.acl} + * property. If this environment variable is set to {@code true}, a + * privileged Docker container will be used if allowed. No other value is + * allowed, so the environment variable should be left unset rather than + * setting it to false. + * + * + * {@code YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS} adds + * additional volume mounts to the Docker container. The value of the + * environment variable should be a comma-separated list of mounts. + * All such mounts must be given as {@code source:dest}, where the + * source is an absolute path that is not a symlink and that points to a + * localized resource. + * +* {code} Maybe we should move this section into the DockerLinuxContainerRuntime.java and replace the section I highlighted with links to the actual runtime classes(DefaultLinuxContainerRuntime, DelegatingLinuxContainerRuntime, DockerLinuxContainerRuntime)? The rest of it looks good to me. > LinuxContainerExecutor needs Javadocs > - > > Key: YARN-5455 > URL: https://issues.apache.org/jira/browse/YARN-5455 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: YARN-5455.001.patch, YARN-5455.002.patch > > > 'Nuff said. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5451) Container localizers that hang are not cleaned up
[ https://issues.apache.org/jira/browse/YARN-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406159#comment-15406159 ] Varun Vasudev commented on YARN-5451: - bq. The localizer should be tracked like containers are tracked so we can control them like we can control containers. +1. This really becomes a problem for containers with large sized resources(like docker). We have to be able to kill a localizer. > Container localizers that hang are not cleaned up > - > > Key: YARN-5451 > URL: https://issues.apache.org/jira/browse/YARN-5451 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe > > I ran across an old, rogue process on one of our nodes. It apparently was a > container localizer that somehow entered an infinite loop during startup. > The NM never cleaned up this broken localizer, so it happily ran forever. > The NM needs to do a better job of tracking localizers, including killing > them if they appear to be hung/broken. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[ https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406144#comment-15406144 ] Hadoop QA commented on YARN-5333: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 54s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 9s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 47m 25s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMHA | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821849/YARN-5333.07.patch | | JIRA Issue | YARN-5333 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 71a03eb4b485 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 2d82276 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/12627/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit test logs | https://builds.apache.org/job/PreCommit-YARN-Build/12627/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12627/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12627/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Some recovered apps a
[jira] [Comment Edited] (YARN-5430) Get container's ip and host from NM
[ https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406094#comment-15406094 ] Varun Vasudev edited comment on YARN-5430 at 8/3/16 4:04 PM: - Thanks for the patch [~jianhe]! 1) {code} + @Private + @Unstable + public abstract void setIp(String ip); + {code} We can’t return just one ip. In the non-docker scenario, multiple ips for a host are common and Docker itself has support for multiple ips for a container. We don’t currently support that but it’ll probably be required for us to support soon. Similarly we should change getIpAndHost and split them up into getIps and getHost calls. 2) {code} + optional string ip = 7; + optional string host = 8; {code} Instead of named variables - can we make this a bit more generic and add support for a simple list of string->list? Then the next time we need to send a new variable, we don’t need to mess around with the protobuf pieces. Something like {code} message StringListMapProto { required string key = 1; optional repeated string value = 2; } {code} and in ContainerStatusProto {code} optional repeated StringListMapProto attributes_map = 7; {code} 3) {code} + public String[] getIpAndHost(Container container) { +return null; + } {code} The default implementation shouldn’t return null. We should return the host name and ip. 4) {code} + public void setIpAndHost(String[] ipAndHost) { +this.ip = ipAndHost[0]; +this.host = ipAndHost[1]; + } {code} This may become a problem if the size of the array is more than 2(multiple ips) 5) {code} + @Override + public String[] getIpAndHost(Container container) { +String[] ipAndHost = new String[2]; +try { + InetAddress address = InetAddress.getLocalHost(); + ipAndHost[0] = address.getHostAddress(); + ipAndHost[1] = address.getHostName(); +} catch (UnknownHostException e) { + LOG.error("Unable to get Local hostname and ip for " + container + .getContainerId()); +} +return ipAndHost; + } } {code} Instead of this, we should just use NetUtils.getLocalHostname() and NetUtils.getIPs. we’ll have to modify getIps() because today it expects a subnet and we want all ips irrespective of subnet. 6) {code} + LOG.info("Docker inspect output for " + containerId + ": " + output); {code} Change log level to debug? 7) {code} +// Be sure to not use space in the argument, otherwise the +// extract_values_delim method in container-executor binary +// cannot parse the arguments correctly. +super.addCommandArguments("--format='{{range.NetworkSettings.Networks}}" ++ "{{.IPAddress}}{{end}},{{.Config.Hostname}}'"); +super.addCommandArguments(containerName); +return this; {code} Ugh. This is an ugly one. I thought there needed to be a space between range and NetworkSettings because the code passed is meant to be a go template. Use this instead - {code}--format '{{range(.NetworkSettings.Networks)}}{{.IPAddress}},{{end}}{{.Config.Hostname}}'{code}. The output will look like - {code}172.17.0.2,192.168.127.128,c399439fbda9{code} 8) {code} + if (docker_binary == NULL) { +docker_binary = "docker"; + } {code} Can we make this a common function - the same code is in launch_docker_container_as_user? was (Author: vvasudev): Thanks for the patch [~jianhe]! 1) {code} + @Private + @Unstable + public abstract void setIp(String ip); + {code} We can’t return just one ip. In the non-docker scenario, multiple ips for a host are common and Docker itself has support for multiple ips for a container. We don’t currently support that but it’ll probably be required for us to support soon. Similarly we should change getIpAndHost and split them up into getIps and getHost calls. 2) {code} + optional string ip = 7; + optional string host = 8; {code} Instead of named variables - can we make this a bit more generic and add support for a simple list of string->list? Then the next time we need to send a new variable, we don’t need to mess around with the protobuf pieces. Something like {code} message StringListMapProto { required string key = 1; optional repeated string value = 2; } {code} and in ContainerStatusProto {code} optional repeated StringListMapProto attributes_map = 7; {code} 3) {code} + public String[] getIpAndHost(Container container) { +return null; + } {code} The default implementation shouldn’t return null. We should return the host name and ip. 4) {code} + public void setIpAndHost(String[] ipAndHost) { +this.ip = ipAndHost[0]; +this.host = ipAndHost[1]; + } {code} This may become a problem if the size of the array is more than 2(multiple ips) 5) {code} + @Override + public String[] getIpAndHost(Container container) { +String[] ipAndHost = new String[2]; +try { + InetAddress address = InetAddress.getLocalHost(); + i
[jira] [Updated] (YARN-5453) FairScheduler#update may skip update demand resource of child queue/app if current demand reached maxResource
[ https://issues.apache.org/jira/browse/YARN-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandflee updated YARN-5453: --- Attachment: YARN-5453.01.patch > FairScheduler#update may skip update demand resource of child queue/app if > current demand reached maxResource > - > > Key: YARN-5453 > URL: https://issues.apache.org/jira/browse/YARN-5453 > Project: Hadoop YARN > Issue Type: Bug >Reporter: sandflee >Assignee: sandflee > Attachments: YARN-5453.01.patch > > > {code} > demand = Resources.createResource(0); > for (FSQueue childQueue : childQueues) { > childQueue.updateDemand(); > Resource toAdd = childQueue.getDemand(); > demand = Resources.add(demand, toAdd); > demand = Resources.componentwiseMin(demand, maxRes); > if (Resources.equals(demand, maxRes)) { > break; > } > } > {code} > if one singe queue's demand resource exceed maxRes, the other queue's demand > resource will not update. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5430) Get container's ip and host from NM
[ https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406094#comment-15406094 ] Varun Vasudev edited comment on YARN-5430 at 8/3/16 3:46 PM: - Thanks for the patch [~jianhe]! 1) {code} + @Private + @Unstable + public abstract void setIp(String ip); + {code} We can’t return just one ip. In the non-docker scenario, multiple ips for a host are common and Docker itself has support for multiple ips for a container. We don’t currently support that but it’ll probably be required for us to support soon. Similarly we should change getIpAndHost and split them up into getIps and getHost calls. 2) {code} + optional string ip = 7; + optional string host = 8; {code} Instead of named variables - can we make this a bit more generic and add support for a simple list of string->list? Then the next time we need to send a new variable, we don’t need to mess around with the protobuf pieces. Something like {code} message StringListMapProto { required string key = 1; optional repeated string value = 2; } {code} and in ContainerStatusProto {code} optional repeated StringListMapProto attributes_map = 7; {code} 3) {code} + public String[] getIpAndHost(Container container) { +return null; + } {code} The default implementation shouldn’t return null. We should return the host name and ip. 4) {code} + public void setIpAndHost(String[] ipAndHost) { +this.ip = ipAndHost[0]; +this.host = ipAndHost[1]; + } {code} This may become a problem if the size of the array is more than 2(multiple ips) 5) {code} + @Override + public String[] getIpAndHost(Container container) { +String[] ipAndHost = new String[2]; +try { + InetAddress address = InetAddress.getLocalHost(); + ipAndHost[0] = address.getHostAddress(); + ipAndHost[1] = address.getHostName(); +} catch (UnknownHostException e) { + LOG.error("Unable to get Local hostname and ip for " + container + .getContainerId()); +} +return ipAndHost; + } } {code} Instead of this, we should just use NetUtils.getLocalHostname() and NetUtils.getIPs. we’ll have to modify getIps() because today it expects a subnet and we want all ips irrespective of subnet. 6) {code} + LOG.info("Docker inspect output for " + containerId + ": " + output); {code} Change log level to debug? 7) {code} +// Be sure to not use space in the argument, otherwise the +// extract_values_delim method in container-executor binary +// cannot parse the arguments correctly. +super.addCommandArguments("--format='{{range.NetworkSettings.Networks}}" ++ "{{.IPAddress}}{{end}},{{.Config.Hostname}}'"); +super.addCommandArguments(containerName); +return this; {code} Ugh. This is an ugly one. I thought there needed to be a space between range and NetworkSettings because the code passed is meant to be a go template. Use this instead - {code}--format='{{range(.NetworkSettings.Networks)}}{code} 8) {code} + if (docker_binary == NULL) { +docker_binary = "docker"; + } {code} Can we make this a common function - the same code is in launch_docker_container_as_user? was (Author: vvasudev): Thanks for the patch [~jianhe]! 1) {code} + @Private + @Unstable + public abstract void setIp(String ip); + {code} We can’t return just one ip. In the non-docker scenario, multiple ips for a host are common and Docker itself has support for multiple ips for a container. We don’t currently support that but it’ll probably be required for us to support soon. Similarly we should change getIpAndHost and split them up into getIps and getHost calls. 2) {code} + optional string ip = 7; + optional string host = 8; {code} Instead of named variables - can we make this a bit more generic and add support for a simple list of string->list? Then the next time we need to send a new variable, we don’t need to mess around with the protobuf pieces. Something like {code} message StringListMapProto { required string key = 1; optional repeated string value = 2; } and in ContainerStatusProto {code} optional repeated StringListMapProto attributes_map = 7; {code} 3) {code} + public String[] getIpAndHost(Container container) { +return null; + } {code} The default implementation shouldn’t return null. We should return the host name and ip. 4) {code} + public void setIpAndHost(String[] ipAndHost) { +this.ip = ipAndHost[0]; +this.host = ipAndHost[1]; + } {code} This may become a problem if the size of the array is more than 2(multiple ips) 5) {code} + @Override + public String[] getIpAndHost(Container container) { +String[] ipAndHost = new String[2]; +try { + InetAddress address = InetAddress.getLocalHost(); + ipAndHost[0] = address.getHostAddress(); + ipAndHost[1] = address.getHostName(); +} catch (UnknownHostException e) { +
[jira] [Commented] (YARN-5430) Get container's ip and host from NM
[ https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406094#comment-15406094 ] Varun Vasudev commented on YARN-5430: - Thanks for the patch [~jianhe]! 1) {code} + @Private + @Unstable + public abstract void setIp(String ip); + {code} We can’t return just one ip. In the non-docker scenario, multiple ips for a host are common and Docker itself has support for multiple ips for a container. We don’t currently support that but it’ll probably be required for us to support soon. Similarly we should change getIpAndHost and split them up into getIps and getHost calls. 2) {code} + optional string ip = 7; + optional string host = 8; {code} Instead of named variables - can we make this a bit more generic and add support for a simple list of string->list? Then the next time we need to send a new variable, we don’t need to mess around with the protobuf pieces. Something like {code} message StringListMapProto { required string key = 1; optional repeated string value = 2; } and in ContainerStatusProto {code} optional repeated StringListMapProto attributes_map = 7; {code} 3) {code} + public String[] getIpAndHost(Container container) { +return null; + } {code} The default implementation shouldn’t return null. We should return the host name and ip. 4) {code} + public void setIpAndHost(String[] ipAndHost) { +this.ip = ipAndHost[0]; +this.host = ipAndHost[1]; + } {code} This may become a problem if the size of the array is more than 2(multiple ips) 5) {code} + @Override + public String[] getIpAndHost(Container container) { +String[] ipAndHost = new String[2]; +try { + InetAddress address = InetAddress.getLocalHost(); + ipAndHost[0] = address.getHostAddress(); + ipAndHost[1] = address.getHostName(); +} catch (UnknownHostException e) { + LOG.error("Unable to get Local hostname and ip for " + container + .getContainerId()); +} +return ipAndHost; + } } {code} Instead of this, we should just use NetUtils.getLocalHostname() and NetUtils.getIPs. we’ll have to modify getIps() because today it expects a subnet and we want all ips irrespective of subnet. 6) {code} + LOG.info("Docker inspect output for " + containerId + ": " + output); {code} Change log level to debug? 7) {code} +// Be sure to not use space in the argument, otherwise the +// extract_values_delim method in container-executor binary +// cannot parse the arguments correctly. +super.addCommandArguments("--format='{{range.NetworkSettings.Networks}}" ++ "{{.IPAddress}}{{end}},{{.Config.Hostname}}'"); +super.addCommandArguments(containerName); +return this; {code} Ugh. This is an ugly one. I thought there needed to be a space between range and NetworkSettings because the code passed is meant to be a go template. Use this instead - --format='{{range(.NetworkSettings.Networks)}} 8) {code} + if (docker_binary == NULL) { +docker_binary = "docker"; + } {code} Can we make this a common function - the same code is in launch_docker_container_as_user? > Get container's ip and host from NM > --- > > Key: YARN-5430 > URL: https://issues.apache.org/jira/browse/YARN-5430 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5430.1.patch, YARN-5430.2.patch, YARN-5430.3.patch > > > In YARN-4757, we introduced a DNS mechanism for containers. That's based on > the assumption that we can get the container's ip and host information and > store it in the registry-service. This jira aims to get the container's ip > and host from the NM, primarily docker container -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[ https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406060#comment-15406060 ] Jun Gong commented on YARN-5333: Thanks [~jianhe]. Attach a new patch to address above comments. It also fix the checkstyle error. > Some recovered apps are put into default queue when RM HA > - > > Key: YARN-5333 > URL: https://issues.apache.org/jira/browse/YARN-5333 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: YARN-5333.01.patch, YARN-5333.02.patch, > YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch, > YARN-5333.06.patch, YARN-5333.07.patch > > > Enable RM HA and use FairScheduler, > {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false, > {{yarn.scheduler.fair.user-as-default-queue}} is set to false. > Reproduce steps: > 1. Start two RMs. > 2. After RMs are running, change both RM's file > {{etc/hadoop/fair-scheduler.xml}}, then add some queues. > 3. Submit some apps to the new added queues. > 4. Stop the active RM, then the standby RM will transit to active and recover > apps. > However the new active RM will put recovered apps into default queue because > it might have not loaded the new {{fair-scheduler.xml}}. We need call > {{initScheduler}} before start active services or bring {{refreshAll()}} in > front of {{rm.transitionToActive()}}. *It seems it is also important for > other scheduler*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5333) Some recovered apps are put into default queue when RM HA
[ https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Gong updated YARN-5333: --- Attachment: YARN-5333.07.patch > Some recovered apps are put into default queue when RM HA > - > > Key: YARN-5333 > URL: https://issues.apache.org/jira/browse/YARN-5333 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jun Gong >Assignee: Jun Gong > Attachments: YARN-5333.01.patch, YARN-5333.02.patch, > YARN-5333.03.patch, YARN-5333.04.patch, YARN-5333.05.patch, > YARN-5333.06.patch, YARN-5333.07.patch > > > Enable RM HA and use FairScheduler, > {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false, > {{yarn.scheduler.fair.user-as-default-queue}} is set to false. > Reproduce steps: > 1. Start two RMs. > 2. After RMs are running, change both RM's file > {{etc/hadoop/fair-scheduler.xml}}, then add some queues. > 3. Submit some apps to the new added queues. > 4. Stop the active RM, then the standby RM will transit to active and recover > apps. > However the new active RM will put recovered apps into default queue because > it might have not loaded the new {{fair-scheduler.xml}}. We need call > {{initScheduler}} before start active services or bring {{refreshAll()}} in > front of {{rm.transitionToActive()}}. *It seems it is also important for > other scheduler*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5079) [Umbrella] Native YARN framework layer for services and beyond
[ https://issues.apache.org/jira/browse/YARN-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406049#comment-15406049 ] Arun Suresh commented on YARN-5079: --- We were going thru SLIDER-787. With regard to the state of slider as it is today, especially with regard to upgarde, we had a couple of questions/clarifications: # Every YARN Container started by the Slider AM consists of an Agent + the Component itself. # Given 1) is true, I assume, the startContainers call to the NM would actually be starting the Agent (as the root process) and the agent starts the Component process. # Given 1) and 2), I assume component ‘upgrade’ is a yarn out-of-band protocol between the slider AM and the Agent where the component process is brought down and restarted. This also implies that currently the AM does not lose the allocation of the actual Container (unless the Agent goes down) # Is a Slider upgrade (AM + Agent) supported currently? I assume if only AM needs to be upgraded, then work preserving AM restart must be enabled to allow the existing containers to bind to the new AM. IF the Agent itself needs to upgraded, I guess there is no other option but to lose the Container allocations (which I guess YARN-4876 might solve ?) # From the docs, it also looks like the actual coordination / orchestration of the upgrade is currently not the responsibility of Slider: An external tool can explicitly decide which container (group of containers / roles) to upgrade, and can issue the necessary Slider commands. Right? # As per the discussions on this JIRA, the proposal is to NOT include the Slider agent in the YARN merge. Is there a plan to provide a similar Agent like component within YARN? We would still need an Agent like component that can (re)configure and (re)start processes as well as doing fancy stuff like port allocations etc. Are you guys planning on opening up JIRAs for the purpose (are they already there)? > [Umbrella] Native YARN framework layer for services and beyond > -- > > Key: YARN-5079 > URL: https://issues.apache.org/jira/browse/YARN-5079 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > > (See overview doc at YARN-4692, modifying and copy-pasting some of the > relevant pieces and sub-section 3.3.1 to track the specific sub-item.) > (This is a companion to YARN-4793 in our effort to simplify the entire story, > but focusing on APIs) > So far, YARN by design has restricted itself to having a very low-level API > that can support any type of application. Frameworks like Apache Hadoop > MapReduce, Apache Tez, Apache Spark, Apache REEF, Apache Twill, Apache Helix > and others ended up exposing higher level APIs that end-users can directly > leverage to build their applications on top of YARN. On the services side, > Apache Slider has done something similar. > With our current attention on making services first-class and simplified, > it's time to take a fresh look at how we can make Apache Hadoop YARN support > services well out of the box. Beyond the functionality that I outlined in the > previous sections in the doc on how NodeManagers can be enhanced to help > services, the biggest missing piece is the framework itself. There is a lot > of very important functionality that a services' framework can own together > with YARN in executing services end-to-end. > In this JIRA I propose we look at having a native Apache Hadoop framework for > running services natively on YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5451) Container localizers that hang are not cleaned up
[ https://issues.apache.org/jira/browse/YARN-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406001#comment-15406001 ] Jason Lowe commented on YARN-5451: -- No, rather it's more an issue that there's no concept of heartbeat timeouts coming from the container localizer. The localization service starts a localizer then expects that localizer to phone in via heartbeats to control it. It doesn't track the pid or any other way to normally control it. The problem with using a timeout interval at the Shell level is that it puts a hard limit on how long to localize something, and there may be cases where we want to keep localizing because good progress is being made (e.g.: large docker container download). The other issue is that during NM restarts we no longer have a Shell instance associated with those processes since they are no longer children of the new NM process. IMHO the clear indication something is amiss is that startLocalizer returns void. There's no pid or cookie of some kind to associate with the localizer instance. It's fire-and-forget. Once we start the localizer we can't stop it without it voluntarily heartbeating, and if the localizer goes rogue and refuses to heartbeat or respond properly to heartbeat commands the NM can't force it to stop like it can with normal containers. The localizer should be tracked like containers are tracked so we can control them like we can control containers. > Container localizers that hang are not cleaned up > - > > Key: YARN-5451 > URL: https://issues.apache.org/jira/browse/YARN-5451 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe > > I ran across an old, rogue process on one of our nodes. It apparently was a > container localizer that somehow entered an infinite loop during startup. > The NM never cleaned up this broken localizer, so it happily ran forever. > The NM needs to do a better job of tracking localizers, including killing > them if they appear to be hung/broken. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[ https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-5469: -- Attachment: YARN-5469.001.patch Attaching patch to increase timeout to 10 seconds. > Increase timeout of TestAmFilter.testFilter > --- > > Key: YARN-5469 > URL: https://issues.apache.org/jira/browse/YARN-5469 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Minor > Attachments: YARN-5469.001.patch > > > Timeout is currently only 1 second. Saw a timeout failure > {noformat} > java.lang.Exception: test timed out after 1000 milliseconds > at java.util.zip.ZipFile.getEntry(Native Method) > at java.util.zip.ZipFile.getEntry(ZipFile.java:311) > at java.util.jar.JarFile.getEntry(JarFile.java:240) > at java.util.jar.JarFile.getJarEntry(JarFile.java:223) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841) > at sun.misc.URLClassPath.getResource(URLClassPath.java:199) > at java.net.URLClassLoader$1.run(URLClassLoader.java:364) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java
[jira] [Moved] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[ https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne moved HADOOP-13462 to YARN-5469: --- Key: YARN-5469 (was: HADOOP-13462) Project: Hadoop YARN (was: Hadoop Common) > Increase timeout of TestAmFilter.testFilter > --- > > Key: YARN-5469 > URL: https://issues.apache.org/jira/browse/YARN-5469 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Priority: Minor > > Timeout is currently only 1 second. Saw a timeout failure > {noformat} > java.lang.Exception: test timed out after 1000 milliseconds > at java.util.zip.ZipFile.getEntry(Native Method) > at java.util.zip.ZipFile.getEntry(ZipFile.java:311) > at java.util.jar.JarFile.getEntry(JarFile.java:240) > at java.util.jar.JarFile.getJarEntry(JarFile.java:223) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841) > at sun.misc.URLClassPath.getResource(URLClassPath.java:199) > at java.net.URLClassLoader$1.run(URLClassLoader.java:364) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1
[jira] [Assigned] (YARN-5469) Increase timeout of TestAmFilter.testFilter
[ https://issues.apache.org/jira/browse/YARN-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger reassigned YARN-5469: - Assignee: Eric Badger > Increase timeout of TestAmFilter.testFilter > --- > > Key: YARN-5469 > URL: https://issues.apache.org/jira/browse/YARN-5469 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Minor > > Timeout is currently only 1 second. Saw a timeout failure > {noformat} > java.lang.Exception: test timed out after 1000 milliseconds > at java.util.zip.ZipFile.getEntry(Native Method) > at java.util.zip.ZipFile.getEntry(ZipFile.java:311) > at java.util.jar.JarFile.getEntry(JarFile.java:240) > at java.util.jar.JarFile.getJarEntry(JarFile.java:223) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:841) > at sun.misc.URLClassPath.getResource(URLClassPath.java:199) > at java.net.URLClassLoader$1.run(URLClassLoader.java:364) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:455) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:367) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:360) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.d
[jira] [Commented] (YARN-5456) container-executor support for FreeBSD, NetBSD, and others if conf path is absolute
[ https://issues.apache.org/jira/browse/YARN-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405911#comment-15405911 ] Allen Wittenauer commented on YARN-5456: Thanks! > container-executor support for FreeBSD, NetBSD, and others if conf path is > absolute > --- > > Key: YARN-5456 > URL: https://issues.apache.org/jira/browse/YARN-5456 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, security >Affects Versions: 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: security > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5456.00.patch, YARN-5456.01.patch > > > YARN-5121 fixed quite a few portability issues, but it also changed how it > determines it's location to be very operating specific for security reasons. > We should add support for FreeBSD to unbreak it's ports entry, NetBSD (the > sysctl options are just in a different order), and for operating systems that > do not have a defined method, an escape hatch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405857#comment-15405857 ] Ying Zhang commented on YARN-5287: -- Thanks [~Naganarasimha] and [~vvasudev]. Comments addressed and patch updated. Please have a look and kindly help to commit it. > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch, YARN-5287.005.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405835#comment-15405835 ] Shane Kumpf commented on YARN-5428: --- Just to add that in the YARN model, we run the docker commands as root and hence the file needs to be owned by root with 400 permissions minimum. YARN is also not aware of the actual username and password, only where the client config is stored. The administrator or user would need to run "docker login" interactively at least once to generate the config.json file and distribute it across the cluster if they wanted to leverage the client config for credentials. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch, YARN-5428.004.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5334) [YARN-3368] Introduce REFRESH button in various UI pages
[ https://issues.apache.org/jira/browse/YARN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405821#comment-15405821 ] Sunil G commented on YARN-5334: --- Thanks [~Sreenath] for the patch and for working on same. Its really a nice feature and helpful add-on while using the UI. Few comments from my side: 1. In {{cluster-overview.js}} , I think we also need to unload *yarn-app* as it will get fresh set of RUNNING apps. 2. Also in {{yarn-apps.js}}, we can unload *yarn-app*. 3. In {{yarn-nodes.js}}, we can unload *yarn-rm-node* 4. in {{yarn-queue-apps.js}}, we can unload *yarn-app*. 5. In {{yarn-queue.js}}, could we remove the unwanted console logs which prints all items. Similarly in {{yarn-queues.js}} 6. {{Refresh}} if we can have custom button which can take property like {{btn-sm btn-primary}}, it ll be helpful. Yes, we can remove whole line in various templates, but its a big changes in all routes (for the common action) and templates. We could come with an optimization and improvement ticket later for same. I also did an initial sanity tests. Looks good. Will do some more test and update if I find some issues. > [YARN-3368] Introduce REFRESH button in various UI pages > > > Key: YARN-5334 > URL: https://issues.apache.org/jira/browse/YARN-5334 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Sunil G >Assignee: Sreenath Somarajapuram > Attachments: YARN-5334-YARN-3368-0001.patch > > > It will be better if we have a common Refresh button in all pages to get the > latest information in all tables such as apps/nodes/queue etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5334) [YARN-3368] Introduce REFRESH button in various UI pages
[ https://issues.apache.org/jira/browse/YARN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405776#comment-15405776 ] Hadoop QA commented on YARN-5334: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 3s {color} | {color:red} Docker failed to build yetus/hadoop:6d3a5f5. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821818/YARN-5334-YARN-3368-0001.patch | | JIRA Issue | YARN-5334 | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12626/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > [YARN-3368] Introduce REFRESH button in various UI pages > > > Key: YARN-5334 > URL: https://issues.apache.org/jira/browse/YARN-5334 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Sunil G >Assignee: Sreenath Somarajapuram > Attachments: YARN-5334-YARN-3368-0001.patch > > > It will be better if we have a common Refresh button in all pages to get the > latest information in all tables such as apps/nodes/queue etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5334) [YARN-3368] Introduce REFRESH button in various UI pages
[ https://issues.apache.org/jira/browse/YARN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated YARN-5334: - Attachment: YARN-5334-YARN-3368-0001.patch [~sunilg] [~leftnoteasy] Please help. > [YARN-3368] Introduce REFRESH button in various UI pages > > > Key: YARN-5334 > URL: https://issues.apache.org/jira/browse/YARN-5334 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Sunil G >Assignee: Sreenath Somarajapuram > Attachments: YARN-5334-YARN-3368-0001.patch > > > It will be better if we have a common Refresh button in all pages to get the > latest information in all tables such as apps/nodes/queue etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4602) Simple and Scalable Message Service for YARN application
[ https://issues.apache.org/jira/browse/YARN-4602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405770#comment-15405770 ] Arun Suresh commented on YARN-4602: --- [~djp], wondering if you've taken a look at apache REEF. It uses an event driven framework called https://reef.apache.org/wake.html that also supports messaging. It uses Netty under the hood. > Simple and Scalable Message Service for YARN application > > > Key: YARN-4602 > URL: https://issues.apache.org/jira/browse/YARN-4602 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications, resourcemanager >Reporter: Junping Du >Assignee: Junping Du > > We are proposing to support MR AM restart with work preserving in > MAPREDUCE-6608 (https://issues.apache.org/jira/browse/MAPREDUCE-6608) that > when AM get failed for some reason, the inflight tasks will keep > running/pending until new AM attempt comes back to continue. One of > prerequisite is tasks should know where the new AM attempt get launched so > TaskUmbilicalProtocol can get retry between clients and new server. > There could be the same requirement for other applications running on YARN > too. Some application decide to handle message delivery itself, e.g. Long > running services can leverage Slider agent to notify messages back and forth. > However, vanilla applications on YARN is hard to achieve this because Hadoop > RPC mechanism essentially is a single way of communication. Although two > directions mechanism like heartbeats (between NM-RM or AM-RM) can get built > on top of it, it make less sense to build the same mechanism between AM and > its application containers - or it need to handle massive of client > connections in AM which could be the new bottleneck for scalability and very > complicated in state maintaining. Instead, we need a new message mechanism > that is simple and scalable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5460) Change container runtime type logging in DelegatingLinuxContainerRuntime to debug
[ https://issues.apache.org/jira/browse/YARN-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405767#comment-15405767 ] Shane Kumpf commented on YARN-5460: --- Thanks, [~vvasudev]! > Change container runtime type logging in DelegatingLinuxContainerRuntime to > debug > - > > Key: YARN-5460 > URL: https://issues.apache.org/jira/browse/YARN-5460 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf >Priority: Trivial > Fix For: 2.9.0 > > Attachments: YARN-5460.001.patch > > > When a container is reacquired, signalContainer is called via > DelegatingLinuxContainerRuntime every second to validate that the container > is still running, until the container completes or is killed, resulting in > many of the following log entries: > {code} > 2016-08-01 15:28:14,529 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: > Using container runtime: DefaultLinuxContainerRuntime > 2016-08-01 15:28:14,533 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: > Using container runtime: DockerLinuxContainerRuntime > 2016-08-01 15:28:15,537 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: > Using container runtime: DefaultLinuxContainerRuntime > 2016-08-01 15:28:15,540 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: > Using container runtime: DockerLinuxContainerRuntime > 2016-08-01 15:28:16,550 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: > Using container runtime: DefaultLinuxContainerRuntime > 2016-08-01 15:28:16,553 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: > Using container runtime: DockerLinuxContainerRuntime > 2016-08-01 15:28:17,561 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: > Using container runtime: DefaultLinuxContainerRuntime > 2016-08-01 15:28:17,561 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: > Using container runtime: DockerLinuxContainerRuntime > 2016-08-01 15:28:18,570 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: > Using container runtime: DefaultLinuxContainerRuntime > 2016-08-01 15:28:18,570 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: > Using container runtime: DockerLinuxContainerRuntime > {code} > The following should be changed to debug: > {code}if (LOG.isInfoEnabled()) { > LOG.info("Using container runtime: " + runtime.getClass() > .getSimpleName()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
[ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405755#comment-15405755 ] Shane Kumpf commented on YARN-5428: --- To be clear, this patch is enabling the ability to set the docker client's configuration directory. It is up to the administrator if they want to store credentials in that configuration. I'm not advocating a global credential store in any regard. Other client configuration is also stored in config.json. To answer your question though, one scenario where this is useful is to allow YARN to automatically pull the image from a docker private repository as needed. A read-only user can be created at Docker hub and given read access to the images they require. By storing this read-only credential in config.json, and setting the property provided in this patch, the administrator or end user no longer needs to "pre-pull" the image on all machines, which is an administrative burden. As Zhankun points out, it is common that the config is owned by root and perms set to 700, but again, it is up to the administrator how they want to leverage the docker client config. Hope that helps. > Allow for specifying the docker client configuration directory > -- > > Key: YARN-5428 > URL: https://issues.apache.org/jira/browse/YARN-5428 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5428.001.patch, YARN-5428.002.patch, > YARN-5428.003.patch, YARN-5428.004.patch > > > The docker client allows for specifying a configuration directory that > contains the docker client's configuration. It is common to store "docker > login" credentials in this config, to avoid the need to docker login on each > cluster member. > By default the docker client config is $HOME/.docker/config.json on Linux. > However, this does not work with the current container executor user > switching and it may also be desirable to centralize this configuration > beyond the single user's home directory. > Note that the command line arg is for the configuration directory NOT the > configuration file. > This change will be needed to allow YARN to automatically pull images at > localization time or within container executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405751#comment-15405751 ] Hadoop QA commented on YARN-5287: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 2s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 8s {color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 27s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821811/YARN-5287.005.patch | | JIRA Issue | YARN-5287 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux 11e214400a4a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d848184 | | Default Java | 1.8.0_101 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12625/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12625/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch, YARN-5287.005.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h..
[jira] [Updated] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Zhang updated YARN-5287: - Attachment: YARN-5287.005.patch > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch, YARN-5287.005.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5402) Fix NoSuchMethodError in ClusterMetricsInfo
[ https://issues.apache.org/jira/browse/YARN-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405656#comment-15405656 ] Weiwei Yang commented on YARN-5402: --- Hello [~sunilg] Some updates I just tried latest trunk, there is no such issue. I setup a 1 node cluster and tried again with 3368 branch, this did not happen to me as well. (Previously I was setting a 3 nodes cluster, not sure if that matters) I'd like to keep this open for a while in case I am able to reproduce it again. > Fix NoSuchMethodError in ClusterMetricsInfo > --- > > Key: YARN-5402 > URL: https://issues.apache.org/jira/browse/YARN-5402 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Affects Versions: YARN-3368 >Reporter: Weiwei Yang > Attachments: YARN-5402.YARN-3368.001.patch > > > When trying out new UI on a cluster, the index page failed to load because of > error {code}java.lang.NoSuchMethodError: > org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getReservedMB()J{code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org