[jira] [Commented] (YARN-4170) AM need to be notified with priority in AllocateResponse
[ https://issues.apache.org/jira/browse/YARN-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961777#comment-14961777 ] Hudson commented on YARN-4170: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #508 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/508/]) YARN-4170. AM need to be notified with priority in AllocateResponse. (jianhe: rev f9da5cdb2b2dd071fd60fc01ea1edf0f79c0819b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java > AM need to be notified with priority in AllocateResponse > - > > Key: YARN-4170 > URL: https://issues.apache.org/jira/browse/YARN-4170 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.8.0 > > Attachments: 0001-YARN-4170.patch, 0002-YARN-4170.patch, > 0003-YARN-4170.patch, 0004-YARN-4170.patch, 0005-YARN-4170-v2.patch, > 0006-YARN-4170.patch, 0007-YARN-4170.patch > > > As discussed in MAPREDUCE-5870, Application Master need to be notified with > priority in Allocate heartbeat. This will help AM to know the priority and > can update JobStatus when client asks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4162) CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API
[ https://issues.apache.org/jira/browse/YARN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961779#comment-14961779 ] Hudson commented on YARN-4162: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #508 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/508/]) YARN-4162. CapacityScheduler: Add resource usage by partition and queue (wangda: rev 4337b263aa23464a01c316336db2319e2858a2b3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceUsageInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionQueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeLabel.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionResourceUsageInfo.java > CapacityScheduler: Add resource usage by partition and queue capacity by > partition to REST API > -- > > Key: YARN-4162 > URL: https://issues.apache.org/jira/browse/YARN-4162 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Fix For: 2.8.0 > > Attachments: YARN-4162.v1.001.patch, YARN-4162.v2.001.patch, > YARN-4162.v2.002.patch, YARN-4162.v2.003.patch, YARN-4162.v2.004.patch, > YARN-4162.v2.005.patch, restAndJsonOutput.zip > > > When Node Labels are enabled then REST Scheduler Information should also > provide partition specific queue information similar to the existing Web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961778#comment-14961778 ] Hudson commented on YARN-2556: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #508 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/508/]) YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java * hadoop-yarn-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java > Tool to measure the performance of the timeline server > -- > > Key: YARN-2556 > URL: https://issues.apache.org/jira/browse/YARN-2556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Chang Li > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, > YARN-2556.1.patch, YARN-2556.10.patch, YARN-2556.11.patch, > YARN-2556.12.patch, YARN-2556.13.patch, YARN-2556.13.whitespacefix.patch, > YARN-2556.14.patch, YARN-2556.14.whitespacefix.patch, YARN-2556.15.patch, > YARN-2556.2.patch, YARN-2556.3.patch, YARN-2556.4.patch, YARN-2556.5.patch, > YARN-2556.6.patch, YARN-2556.7.patch, YARN-2556.8.patch, YARN-2556.9.patch, > YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch > > > We need to be able to understand the capacity model for the timeline server > to give users the tools they need to deploy a timeline server with the > correct capacity. > I propose we create a mapreduce job that can measure timeline server write > and read performance. Transactions per second, I/O for both read and write > would be a good start. > This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-4068) Support appUpdated event in TimelineV2 to publish details for movetoqueue, change in priority
[ https://issues.apache.org/jira/browse/YARN-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-4068. --- Resolution: Won't Fix Looks like related changes are already in during branch checkin. If any missing parts are there, I will reopen. Thank you. > Support appUpdated event in TimelineV2 to publish details for movetoqueue, > change in priority > - > > Key: YARN-4068 > URL: https://issues.apache.org/jira/browse/YARN-4068 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sunil G >Assignee: Sunil G > > YARN-4044 supports appUpdated event changes to TimelineV1. This jira is to > track and port appUpdated changes in V2 for > - movetoqueue > - updateAppPriority -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961745#comment-14961745 ] Hudson commented on YARN-2556: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2494 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2494/]) YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java * hadoop-yarn-project/CHANGES.txt > Tool to measure the performance of the timeline server > -- > > Key: YARN-2556 > URL: https://issues.apache.org/jira/browse/YARN-2556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Chang Li > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, > YARN-2556.1.patch, YARN-2556.10.patch, YARN-2556.11.patch, > YARN-2556.12.patch, YARN-2556.13.patch, YARN-2556.13.whitespacefix.patch, > YARN-2556.14.patch, YARN-2556.14.whitespacefix.patch, YARN-2556.15.patch, > YARN-2556.2.patch, YARN-2556.3.patch, YARN-2556.4.patch, YARN-2556.5.patch, > YARN-2556.6.patch, YARN-2556.7.patch, YARN-2556.8.patch, YARN-2556.9.patch, > YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch > > > We need to be able to understand the capacity model for the timeline server > to give users the tools they need to deploy a timeline server with the > correct capacity. > I propose we create a mapreduce job that can measure timeline server write > and read performance. Transactions per second, I/O for both read and write > would be a good start. > This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961724#comment-14961724 ] Hudson commented on YARN-2556: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #1281 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1281/]) YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java * hadoop-yarn-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java > Tool to measure the performance of the timeline server > -- > > Key: YARN-2556 > URL: https://issues.apache.org/jira/browse/YARN-2556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Chang Li > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, > YARN-2556.1.patch, YARN-2556.10.patch, YARN-2556.11.patch, > YARN-2556.12.patch, YARN-2556.13.patch, YARN-2556.13.whitespacefix.patch, > YARN-2556.14.patch, YARN-2556.14.whitespacefix.patch, YARN-2556.15.patch, > YARN-2556.2.patch, YARN-2556.3.patch, YARN-2556.4.patch, YARN-2556.5.patch, > YARN-2556.6.patch, YARN-2556.7.patch, YARN-2556.8.patch, YARN-2556.9.patch, > YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch > > > We need to be able to understand the capacity model for the timeline server > to give users the tools they need to deploy a timeline server with the > correct capacity. > I propose we create a mapreduce job that can measure timeline server write > and read performance. Transactions per second, I/O for both read and write > would be a good start. > This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961721#comment-14961721 ] Hudson commented on YARN-2556: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2445 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2445/]) YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java * hadoop-yarn-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java > Tool to measure the performance of the timeline server > -- > > Key: YARN-2556 > URL: https://issues.apache.org/jira/browse/YARN-2556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Chang Li > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, > YARN-2556.1.patch, YARN-2556.10.patch, YARN-2556.11.patch, > YARN-2556.12.patch, YARN-2556.13.patch, YARN-2556.13.whitespacefix.patch, > YARN-2556.14.patch, YARN-2556.14.whitespacefix.patch, YARN-2556.15.patch, > YARN-2556.2.patch, YARN-2556.3.patch, YARN-2556.4.patch, YARN-2556.5.patch, > YARN-2556.6.patch, YARN-2556.7.patch, YARN-2556.8.patch, YARN-2556.9.patch, > YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch > > > We need to be able to understand the capacity model for the timeline server > to give users the tools they need to deploy a timeline server with the > correct capacity. > I propose we create a mapreduce job that can measure timeline server write > and read performance. Transactions per second, I/O for both read and write > would be a good start. > This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4170) AM need to be notified with priority in AllocateResponse
[ https://issues.apache.org/jira/browse/YARN-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961720#comment-14961720 ] Hudson commented on YARN-4170: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2445 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2445/]) YARN-4170. AM need to be notified with priority in AllocateResponse. (jianhe: rev f9da5cdb2b2dd071fd60fc01ea1edf0f79c0819b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java > AM need to be notified with priority in AllocateResponse > - > > Key: YARN-4170 > URL: https://issues.apache.org/jira/browse/YARN-4170 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.8.0 > > Attachments: 0001-YARN-4170.patch, 0002-YARN-4170.patch, > 0003-YARN-4170.patch, 0004-YARN-4170.patch, 0005-YARN-4170-v2.patch, > 0006-YARN-4170.patch, 0007-YARN-4170.patch > > > As discussed in MAPREDUCE-5870, Application Master need to be notified with > priority in Allocate heartbeat. This will help AM to know the priority and > can update JobStatus when client asks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4162) CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API
[ https://issues.apache.org/jira/browse/YARN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961722#comment-14961722 ] Hudson commented on YARN-4162: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2445 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2445/]) YARN-4162. CapacityScheduler: Add resource usage by partition and queue (wangda: rev 4337b263aa23464a01c316336db2319e2858a2b3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceUsageInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionResourceUsageInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeLabel.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionQueueCapacitiesInfo.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java > CapacityScheduler: Add resource usage by partition and queue capacity by > partition to REST API > -- > > Key: YARN-4162 > URL: https://issues.apache.org/jira/browse/YARN-4162 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Fix For: 2.8.0 > > Attachments: YARN-4162.v1.001.patch, YARN-4162.v2.001.patch, > YARN-4162.v2.002.patch, YARN-4162.v2.003.patch, YARN-4162.v2.004.patch, > YARN-4162.v2.005.patch, restAndJsonOutput.zip > > > When Node Labels are enabled then REST Scheduler Information should also > provide partition specific queue information similar to the existing Web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2729) Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup
[
https://issues.apache.org/jira/browse/YARN-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961718#comment-14961718
]
Sunil G commented on YARN-2729:
---
Hi [~Naganarasimha] and [~rohithsharma]
Sorry for coming in late. I have some doubts here.
In {{ScriptBasedNodeLabelsProvider#serviceStop}}, {{shexec}} process is
destroyed and the {{super.stop()}} is invoked. Hence I feel it may still be
possible that {{shexec}} may again get executed (if one timer run happens in
this timeframe) before timer is cancelled from
{{AbstractNodeLabelsProvider#serviceStop}}. We can do {{super.stop()}} first
from ScriptBasedNodeLabelsProvider. But I feel it may not be better, so could
we reset *shexec* to null after attached process is destroyed (may also need a
null check in timetask if we do this). Thoughts?
couple of other comments:
1. ScriptBasedNodeLabelsProvider#serviceInit: {{verifyConfiguredScript}} is
invoked with {{nodeLabelsScript}} which is a class variable. It may not be
needed.
2. I have one doubt in {{fetchLabelsFromScriptOutput}}. {{scriptOutput}} is
split with newline and each line is checked whether its starts with
NODE_LABEL_PARTITION_PATTERN. But if we have multiple lines from input string
which starts with NODE_LABEL_PARTITION_PATTERN, then only last line will be set
to {{nodePartitionLabel}}. I didnt understand this part. Could you please have
a look on this.
> Support script based NodeLabelsProvider Interface in Distributed Node Label
> Configuration Setup
> ---
>
> Key: YARN-2729
> URL: https://issues.apache.org/jira/browse/YARN-2729
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-2729.20141023-1.patch, YARN-2729.20141024-1.patch,
> YARN-2729.20141031-1.patch, YARN-2729.20141120-1.patch,
> YARN-2729.20141210-1.patch, YARN-2729.20150309-1.patch,
> YARN-2729.20150322-1.patch, YARN-2729.20150401-1.patch,
> YARN-2729.20150402-1.patch, YARN-2729.20150404-1.patch,
> YARN-2729.20150517-1.patch, YARN-2729.20150830-1.patch,
> YARN-2729.20150925-1.patch, YARN-2729.20151015-1.patch,
> YARN-2729.20151310-1.patch, YARN-2729.20151310-2.patch
>
>
> Support script based NodeLabelsProvider Interface in Distributed Node Label
> Configuration Setup .
> Miscellaneous Issues :
> # In configurationNodeLabelsProvider instead of taking multiple labels from
> single configuration, we need to support exclusive configuration for
> partition (single label).
> # Proper logging when registration of Node Fails
> # Classloader was not getting reset from custom class loader in
> TestConfigurationNodeLabelsProvider.java which could make test cases fail in
> certain conditions
> # In ResourceTrackerService we need to consider distributed configuration
> only when node labels are enabled. leads to lots of logs in certain conditions
> # NodeLabelsProvider needs to be a interface rather than abstract class
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-2729) Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup
[ https://issues.apache.org/jira/browse/YARN-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G reassigned YARN-2729: - Assignee: Sunil G (was: Naganarasimha G R) > Support script based NodeLabelsProvider Interface in Distributed Node Label > Configuration Setup > --- > > Key: YARN-2729 > URL: https://issues.apache.org/jira/browse/YARN-2729 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Naganarasimha G R >Assignee: Sunil G > Attachments: YARN-2729.20141023-1.patch, YARN-2729.20141024-1.patch, > YARN-2729.20141031-1.patch, YARN-2729.20141120-1.patch, > YARN-2729.20141210-1.patch, YARN-2729.20150309-1.patch, > YARN-2729.20150322-1.patch, YARN-2729.20150401-1.patch, > YARN-2729.20150402-1.patch, YARN-2729.20150404-1.patch, > YARN-2729.20150517-1.patch, YARN-2729.20150830-1.patch, > YARN-2729.20150925-1.patch, YARN-2729.20151015-1.patch, > YARN-2729.20151310-1.patch, YARN-2729.20151310-2.patch > > > Support script based NodeLabelsProvider Interface in Distributed Node Label > Configuration Setup . > Miscellaneous Issues : > # In configurationNodeLabelsProvider instead of taking multiple labels from > single configuration, we need to support exclusive configuration for > partition (single label). > # Proper logging when registration of Node Fails > # Classloader was not getting reset from custom class loader in > TestConfigurationNodeLabelsProvider.java which could make test cases fail in > certain conditions > # In ResourceTrackerService we need to consider distributed configuration > only when node labels are enabled. leads to lots of logs in certain conditions > # NodeLabelsProvider needs to be a interface rather than abstract class -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2729) Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup
[ https://issues.apache.org/jira/browse/YARN-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2729: -- Assignee: Naganarasimha G R (was: Sunil G) > Support script based NodeLabelsProvider Interface in Distributed Node Label > Configuration Setup > --- > > Key: YARN-2729 > URL: https://issues.apache.org/jira/browse/YARN-2729 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: YARN-2729.20141023-1.patch, YARN-2729.20141024-1.patch, > YARN-2729.20141031-1.patch, YARN-2729.20141120-1.patch, > YARN-2729.20141210-1.patch, YARN-2729.20150309-1.patch, > YARN-2729.20150322-1.patch, YARN-2729.20150401-1.patch, > YARN-2729.20150402-1.patch, YARN-2729.20150404-1.patch, > YARN-2729.20150517-1.patch, YARN-2729.20150830-1.patch, > YARN-2729.20150925-1.patch, YARN-2729.20151015-1.patch, > YARN-2729.20151310-1.patch, YARN-2729.20151310-2.patch > > > Support script based NodeLabelsProvider Interface in Distributed Node Label > Configuration Setup . > Miscellaneous Issues : > # In configurationNodeLabelsProvider instead of taking multiple labels from > single configuration, we need to support exclusive configuration for > partition (single label). > # Proper logging when registration of Node Fails > # Classloader was not getting reset from custom class loader in > TestConfigurationNodeLabelsProvider.java which could make test cases fail in > certain conditions > # In ResourceTrackerService we need to consider distributed configuration > only when node labels are enabled. leads to lots of logs in certain conditions > # NodeLabelsProvider needs to be a interface rather than abstract class -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961709#comment-14961709 ] Hudson commented on YARN-2556: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #545 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/545/]) YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) * hadoop-yarn-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java > Tool to measure the performance of the timeline server > -- > > Key: YARN-2556 > URL: https://issues.apache.org/jira/browse/YARN-2556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Chang Li > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, > YARN-2556.1.patch, YARN-2556.10.patch, YARN-2556.11.patch, > YARN-2556.12.patch, YARN-2556.13.patch, YARN-2556.13.whitespacefix.patch, > YARN-2556.14.patch, YARN-2556.14.whitespacefix.patch, YARN-2556.15.patch, > YARN-2556.2.patch, YARN-2556.3.patch, YARN-2556.4.patch, YARN-2556.5.patch, > YARN-2556.6.patch, YARN-2556.7.patch, YARN-2556.8.patch, YARN-2556.9.patch, > YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch > > > We need to be able to understand the capacity model for the timeline server > to give users the tools they need to deploy a timeline server with the > correct capacity. > I propose we create a mapreduce job that can measure timeline server write > and read performance. Transactions per second, I/O for both read and write > would be a good start. > This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3798) ZKRMStateStore shouldn't create new session without occurrance of SESSIONEXPIED
[
https://issues.apache.org/jira/browse/YARN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961683#comment-14961683
]
Tsuyoshi Ozawa commented on YARN-3798:
--
s/It's ready for review/It's ready for committing/
> ZKRMStateStore shouldn't create new session without occurrance of
> SESSIONEXPIED
> ---
>
> Key: YARN-3798
> URL: https://issues.apache.org/jira/browse/YARN-3798
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.0
> Environment: Suse 11 Sp3
>Reporter: Bibin A Chundatt
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: RM.log, YARN-3798-2.7.002.patch,
> YARN-3798-branch-2.6.01.patch, YARN-3798-branch-2.7.002.patch,
> YARN-3798-branch-2.7.003.patch, YARN-3798-branch-2.7.004.patch,
> YARN-3798-branch-2.7.005.patch, YARN-3798-branch-2.7.006.patch,
> YARN-3798-branch-2.7.patch
>
>
> RM going down with NoNode exception during create of znode for appattempt
> *Please find the exception logs*
> {code}
> 2015-06-09 10:09:44,732 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore:
> ZKRMStateStore Session connected
> 2015-06-09 10:09:44,732 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore:
> ZKRMStateStore Session restored
> 2015-06-09 10:09:44,886 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore:
> Exception while executing a ZK operation.
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1405)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1310)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:926)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:923)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1122)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:923)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:937)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createWithRetries(ZKRMStateStore.java:970)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.updateApplicationAttemptStateInternal(ZKRMStateStore.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:260)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:837)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:900)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:895)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
> at java.lang.Thread.run(Thread.java:745)
> 2015-06-09 10:09:44,887 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore: Maxed
> out ZK retries. Giving up!
> 2015-06-09 10:09:44,887 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Error
> updating appAttempt: appattempt_1433764310492_7152_01
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1405)
> at org.apache.zookeeper.ZooKeeper
[jira] [Commented] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961668#comment-14961668 ] Hudson commented on YARN-2556: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #560 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/560/]) YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java * hadoop-yarn-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java > Tool to measure the performance of the timeline server > -- > > Key: YARN-2556 > URL: https://issues.apache.org/jira/browse/YARN-2556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Chang Li > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, > YARN-2556.1.patch, YARN-2556.10.patch, YARN-2556.11.patch, > YARN-2556.12.patch, YARN-2556.13.patch, YARN-2556.13.whitespacefix.patch, > YARN-2556.14.patch, YARN-2556.14.whitespacefix.patch, YARN-2556.15.patch, > YARN-2556.2.patch, YARN-2556.3.patch, YARN-2556.4.patch, YARN-2556.5.patch, > YARN-2556.6.patch, YARN-2556.7.patch, YARN-2556.8.patch, YARN-2556.9.patch, > YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch > > > We need to be able to understand the capacity model for the timeline server > to give users the tools they need to deploy a timeline server with the > correct capacity. > I propose we create a mapreduce job that can measure timeline server write > and read performance. Transactions per second, I/O for both read and write > would be a good start. > This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4170) AM need to be notified with priority in AllocateResponse
[ https://issues.apache.org/jira/browse/YARN-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961663#comment-14961663 ] Hudson commented on YARN-4170: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2493 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2493/]) YARN-4170. AM need to be notified with priority in AllocateResponse. (jianhe: rev f9da5cdb2b2dd071fd60fc01ea1edf0f79c0819b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto > AM need to be notified with priority in AllocateResponse > - > > Key: YARN-4170 > URL: https://issues.apache.org/jira/browse/YARN-4170 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.8.0 > > Attachments: 0001-YARN-4170.patch, 0002-YARN-4170.patch, > 0003-YARN-4170.patch, 0004-YARN-4170.patch, 0005-YARN-4170-v2.patch, > 0006-YARN-4170.patch, 0007-YARN-4170.patch > > > As discussed in MAPREDUCE-5870, Application Master need to be notified with > priority in Allocate heartbeat. This will help AM to know the priority and > can update JobStatus when client asks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4162) CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API
[ https://issues.apache.org/jira/browse/YARN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961664#comment-14961664 ] Hudson commented on YARN-4162: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2493 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2493/]) YARN-4162. CapacityScheduler: Add resource usage by partition and queue (wangda: rev 4337b263aa23464a01c316336db2319e2858a2b3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeLabel.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionResourceUsageInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceUsageInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionQueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java > CapacityScheduler: Add resource usage by partition and queue capacity by > partition to REST API > -- > > Key: YARN-4162 > URL: https://issues.apache.org/jira/browse/YARN-4162 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Fix For: 2.8.0 > > Attachments: YARN-4162.v1.001.patch, YARN-4162.v2.001.patch, > YARN-4162.v2.002.patch, YARN-4162.v2.003.patch, YARN-4162.v2.004.patch, > YARN-4162.v2.005.patch, restAndJsonOutput.zip > > > When Node Labels are enabled then REST Scheduler Information should also > provide partition specific queue information similar to the existing Web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4169) jenkins trunk+java build failed in TestNodeStatusUpdaterForLabels
[
https://issues.apache.org/jira/browse/YARN-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961650#comment-14961650
]
Naganarasimha G R commented on YARN-4169:
-
Any thoughts on the above comment, [~rohithsharma] & [~wangda] ?
> jenkins trunk+java build failed in TestNodeStatusUpdaterForLabels
> -
>
> Key: YARN-4169
> URL: https://issues.apache.org/jira/browse/YARN-4169
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Naganarasimha G R
>Priority: Critical
> Attachments: YARN-4169.v1.001.patch, YARN-4169.v1.002.patch,
> YARN-4169.v1.003.patch
>
>
> Test failing in [[Jenkins build
> 402|https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk-Java8/402/testReport/junit/org.apache.hadoop.yarn.server.nodemanager/TestNodeStatusUpdaterForLabels/testNodeStatusUpdaterForNodeLabels/]
> {code}
> java.lang.NullPointerException: null
> at java.util.HashSet.(HashSet.java:118)
> at
> org.apache.hadoop.yarn.nodelabels.NodeLabelTestBase.assertNLCollectionEquals(NodeLabelTestBase.java:103)
> at
> org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels.testNodeStatusUpdaterForNodeLabels(TestNodeStatusUpdaterForLabels.java:268)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4170) AM need to be notified with priority in AllocateResponse
[ https://issues.apache.org/jira/browse/YARN-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961633#comment-14961633 ] Sunil G commented on YARN-4170: --- Thank you very much [~jianhe] for the review and commit and thank you Rohith for the review. > AM need to be notified with priority in AllocateResponse > - > > Key: YARN-4170 > URL: https://issues.apache.org/jira/browse/YARN-4170 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.8.0 > > Attachments: 0001-YARN-4170.patch, 0002-YARN-4170.patch, > 0003-YARN-4170.patch, 0004-YARN-4170.patch, 0005-YARN-4170-v2.patch, > 0006-YARN-4170.patch, 0007-YARN-4170.patch > > > As discussed in MAPREDUCE-5870, Application Master need to be notified with > priority in Allocate heartbeat. This will help AM to know the priority and > can update JobStatus when client asks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4170) AM need to be notified with priority in AllocateResponse
[ https://issues.apache.org/jira/browse/YARN-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961615#comment-14961615 ] Hudson commented on YARN-4170: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1280 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1280/]) YARN-4170. AM need to be notified with priority in AllocateResponse. (jianhe: rev f9da5cdb2b2dd071fd60fc01ea1edf0f79c0819b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java > AM need to be notified with priority in AllocateResponse > - > > Key: YARN-4170 > URL: https://issues.apache.org/jira/browse/YARN-4170 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.8.0 > > Attachments: 0001-YARN-4170.patch, 0002-YARN-4170.patch, > 0003-YARN-4170.patch, 0004-YARN-4170.patch, 0005-YARN-4170-v2.patch, > 0006-YARN-4170.patch, 0007-YARN-4170.patch > > > As discussed in MAPREDUCE-5870, Application Master need to be notified with > priority in Allocate heartbeat. This will help AM to know the priority and > can update JobStatus when client asks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4162) CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API
[ https://issues.apache.org/jira/browse/YARN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961616#comment-14961616 ] Hudson commented on YARN-4162: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1280 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1280/]) YARN-4162. CapacityScheduler: Add resource usage by partition and queue (wangda: rev 4337b263aa23464a01c316336db2319e2858a2b3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceUsageInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionQueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeLabel.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionResourceUsageInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java > CapacityScheduler: Add resource usage by partition and queue capacity by > partition to REST API > -- > > Key: YARN-4162 > URL: https://issues.apache.org/jira/browse/YARN-4162 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Fix For: 2.8.0 > > Attachments: YARN-4162.v1.001.patch, YARN-4162.v2.001.patch, > YARN-4162.v2.002.patch, YARN-4162.v2.003.patch, YARN-4162.v2.004.patch, > YARN-4162.v2.005.patch, restAndJsonOutput.zip > > > When Node Labels are enabled then REST Scheduler Information should also > provide partition specific queue information similar to the existing Web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961606#comment-14961606 ] Hudson commented on YARN-2556: -- SUCCESS: Integrated in Hadoop-trunk-Commit #8654 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8654/]) YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java * hadoop-yarn-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java > Tool to measure the performance of the timeline server > -- > > Key: YARN-2556 > URL: https://issues.apache.org/jira/browse/YARN-2556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Chang Li > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, > YARN-2556.1.patch, YARN-2556.10.patch, YARN-2556.11.patch, > YARN-2556.12.patch, YARN-2556.13.patch, YARN-2556.13.whitespacefix.patch, > YARN-2556.14.patch, YARN-2556.14.whitespacefix.patch, YARN-2556.15.patch, > YARN-2556.2.patch, YARN-2556.3.patch, YARN-2556.4.patch, YARN-2556.5.patch, > YARN-2556.6.patch, YARN-2556.7.patch, YARN-2556.8.patch, YARN-2556.9.patch, > YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch > > > We need to be able to understand the capacity model for the timeline server > to give users the tools they need to deploy a timeline server with the > correct capacity. > I propose we create a mapreduce job that can measure timeline server write > and read performance. Transactions per second, I/O for both read and write > would be a good start. > This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961578#comment-14961578
]
Hadoop QA commented on YARN-1509:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 18s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 2s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 54s | The applied patch generated 3
new checkstyle issues (total was 79, now 76). |
| {color:green}+1{color} | whitespace | 0m 12s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 40s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 7m 0s | Tests passed in
hadoop-yarn-applications-distributedshell. |
| {color:green}+1{color} | yarn tests | 7m 29s | Tests passed in
hadoop-yarn-client. |
| | | 55m 59s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767154/YARN-1509.6.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f9da5cd |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9467/artifact/patchprocess/diffcheckstylehadoop-yarn-client.txt
|
| hadoop-yarn-applications-distributedshell test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9467/artifact/patchprocess/testrun_hadoop-yarn-applications-distributedshell.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9467/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9467/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9467/console |
This message was automatically generated.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch, YARN-1509.6.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961542#comment-14961542 ] Sangjin Lee commented on YARN-2556: --- The latest patch (v.15) looks good to me. I'll commit it shortly. > Tool to measure the performance of the timeline server > -- > > Key: YARN-2556 > URL: https://issues.apache.org/jira/browse/YARN-2556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Chang Li > Labels: BB2015-05-TBR > Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, > YARN-2556.1.patch, YARN-2556.10.patch, YARN-2556.11.patch, > YARN-2556.12.patch, YARN-2556.13.patch, YARN-2556.13.whitespacefix.patch, > YARN-2556.14.patch, YARN-2556.14.whitespacefix.patch, YARN-2556.15.patch, > YARN-2556.2.patch, YARN-2556.3.patch, YARN-2556.4.patch, YARN-2556.5.patch, > YARN-2556.6.patch, YARN-2556.7.patch, YARN-2556.8.patch, YARN-2556.9.patch, > YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch > > > We need to be able to understand the capacity model for the timeline server > to give users the tools they need to deploy a timeline server with the > correct capacity. > I propose we create a mapreduce job that can measure timeline server write > and read performance. Transactions per second, I/O for both read and write > would be a good start. > This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4162) CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API
[ https://issues.apache.org/jira/browse/YARN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961537#comment-14961537 ] Hudson commented on YARN-4162: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #559 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/559/]) YARN-4162. CapacityScheduler: Add resource usage by partition and queue (wangda: rev 4337b263aa23464a01c316336db2319e2858a2b3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionResourceUsageInfo.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceUsageInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeLabel.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionQueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java > CapacityScheduler: Add resource usage by partition and queue capacity by > partition to REST API > -- > > Key: YARN-4162 > URL: https://issues.apache.org/jira/browse/YARN-4162 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Fix For: 2.8.0 > > Attachments: YARN-4162.v1.001.patch, YARN-4162.v2.001.patch, > YARN-4162.v2.002.patch, YARN-4162.v2.003.patch, YARN-4162.v2.004.patch, > YARN-4162.v2.005.patch, restAndJsonOutput.zip > > > When Node Labels are enabled then REST Scheduler Information should also > provide partition specific queue information similar to the existing Web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4170) AM need to be notified with priority in AllocateResponse
[ https://issues.apache.org/jira/browse/YARN-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961536#comment-14961536 ] Hudson commented on YARN-4170: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #559 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/559/]) YARN-4170. AM need to be notified with priority in AllocateResponse. (jianhe: rev f9da5cdb2b2dd071fd60fc01ea1edf0f79c0819b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java > AM need to be notified with priority in AllocateResponse > - > > Key: YARN-4170 > URL: https://issues.apache.org/jira/browse/YARN-4170 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.8.0 > > Attachments: 0001-YARN-4170.patch, 0002-YARN-4170.patch, > 0003-YARN-4170.patch, 0004-YARN-4170.patch, 0005-YARN-4170-v2.patch, > 0006-YARN-4170.patch, 0007-YARN-4170.patch > > > As discussed in MAPREDUCE-5870, Application Master need to be notified with > priority in Allocate heartbeat. This will help AM to know the priority and > can update JobStatus when client asks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2556) Tool to measure the performance of the timeline server
[
https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961532#comment-14961532
]
Hadoop QA commented on YARN-2556:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 6m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 53s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 33s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 49s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 98m 7s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 116m 28s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestNetworkedJob |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767126/YARN-2556.15.patch |
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 79b8d60 |
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9466/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9466/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9466/console |
This message was automatically generated.
> Tool to measure the performance of the timeline server
> --
>
> Key: YARN-2556
> URL: https://issues.apache.org/jira/browse/YARN-2556
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Jonathan Eagles
>Assignee: Chang Li
> Labels: BB2015-05-TBR
> Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch,
> YARN-2556.1.patch, YARN-2556.10.patch, YARN-2556.11.patch,
> YARN-2556.12.patch, YARN-2556.13.patch, YARN-2556.13.whitespacefix.patch,
> YARN-2556.14.patch, YARN-2556.14.whitespacefix.patch, YARN-2556.15.patch,
> YARN-2556.2.patch, YARN-2556.3.patch, YARN-2556.4.patch, YARN-2556.5.patch,
> YARN-2556.6.patch, YARN-2556.7.patch, YARN-2556.8.patch, YARN-2556.9.patch,
> YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch
>
>
> We need to be able to understand the capacity model for the timeline server
> to give users the tools they need to deploy a timeline server with the
> correct capacity.
> I propose we create a mapreduce job that can measure timeline server write
> and read performance. Transactions per second, I/O for both read and write
> would be a good start.
> This could be done as an example or test job that could be tied into gridmix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4170) AM need to be notified with priority in AllocateResponse
[ https://issues.apache.org/jira/browse/YARN-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961520#comment-14961520 ] Hudson commented on YARN-4170: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #544 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/544/]) YARN-4170. AM need to be notified with priority in AllocateResponse. (jianhe: rev f9da5cdb2b2dd071fd60fc01ea1edf0f79c0819b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java > AM need to be notified with priority in AllocateResponse > - > > Key: YARN-4170 > URL: https://issues.apache.org/jira/browse/YARN-4170 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.8.0 > > Attachments: 0001-YARN-4170.patch, 0002-YARN-4170.patch, > 0003-YARN-4170.patch, 0004-YARN-4170.patch, 0005-YARN-4170-v2.patch, > 0006-YARN-4170.patch, 0007-YARN-4170.patch > > > As discussed in MAPREDUCE-5870, Application Master need to be notified with > priority in Allocate heartbeat. This will help AM to know the priority and > can update JobStatus when client asks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4162) CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API
[ https://issues.apache.org/jira/browse/YARN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961521#comment-14961521 ] Hudson commented on YARN-4162: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #544 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/544/]) YARN-4162. CapacityScheduler: Add resource usage by partition and queue (wangda: rev 4337b263aa23464a01c316336db2319e2858a2b3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionResourceUsageInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeLabel.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionQueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceUsageInfo.java > CapacityScheduler: Add resource usage by partition and queue capacity by > partition to REST API > -- > > Key: YARN-4162 > URL: https://issues.apache.org/jira/browse/YARN-4162 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Fix For: 2.8.0 > > Attachments: YARN-4162.v1.001.patch, YARN-4162.v2.001.patch, > YARN-4162.v2.002.patch, YARN-4162.v2.003.patch, YARN-4162.v2.004.patch, > YARN-4162.v2.005.patch, restAndJsonOutput.zip > > > When Node Labels are enabled then REST Scheduler Information should also > provide partition specific queue information similar to the existing Web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4162) CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API
[ https://issues.apache.org/jira/browse/YARN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961496#comment-14961496 ] Hudson commented on YARN-4162: -- FAILURE: Integrated in Hadoop-trunk-Commit #8653 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8653/]) YARN-4162. CapacityScheduler: Add resource usage by partition and queue (wangda: rev 4337b263aa23464a01c316336db2319e2858a2b3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceUsageInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionQueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/PartitionResourceUsageInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeLabel.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/QueueCapacitiesInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java > CapacityScheduler: Add resource usage by partition and queue capacity by > partition to REST API > -- > > Key: YARN-4162 > URL: https://issues.apache.org/jira/browse/YARN-4162 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Fix For: 2.8.0 > > Attachments: YARN-4162.v1.001.patch, YARN-4162.v2.001.patch, > YARN-4162.v2.002.patch, YARN-4162.v2.003.patch, YARN-4162.v2.004.patch, > YARN-4162.v2.005.patch, restAndJsonOutput.zip > > > When Node Labels are enabled then REST Scheduler Information should also > provide partition specific queue information similar to the existing Web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4170) AM need to be notified with priority in AllocateResponse
[ https://issues.apache.org/jira/browse/YARN-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961495#comment-14961495 ] Hudson commented on YARN-4170: -- FAILURE: Integrated in Hadoop-trunk-Commit #8653 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8653/]) YARN-4170. AM need to be notified with priority in AllocateResponse. (jianhe: rev f9da5cdb2b2dd071fd60fc01ea1edf0f79c0819b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java > AM need to be notified with priority in AllocateResponse > - > > Key: YARN-4170 > URL: https://issues.apache.org/jira/browse/YARN-4170 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.8.0 > > Attachments: 0001-YARN-4170.patch, 0002-YARN-4170.patch, > 0003-YARN-4170.patch, 0004-YARN-4170.patch, 0005-YARN-4170-v2.patch, > 0006-YARN-4170.patch, 0007-YARN-4170.patch > > > As discussed in MAPREDUCE-5870, Application Master need to be notified with > priority in Allocate heartbeat. This will help AM to know the priority and > can update JobStatus when client asks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MENG DING updated YARN-1509: Attachment: YARN-1509.6.patch Attaching new patch that address the following issues: * Combine increase/decrease requests into one method * Combine increase/decrease callback methods into one method * Deprecate the CallbackHandler interface and other related methods * Remove pending change requests of a container when that container is released, or is completed * Update related test cases * Add a test case to test recovery resource change requests on RM restart > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch, YARN-1509.5.patch, YARN-1509.6.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4162) CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API
[ https://issues.apache.org/jira/browse/YARN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4162: - Summary: CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API (was: Scheduler info in REST, is currently not displaying partition specific queue information similar to UI) > CapacityScheduler: Add resource usage by partition and queue capacity by > partition to REST API > -- > > Key: YARN-4162 > URL: https://issues.apache.org/jira/browse/YARN-4162 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: YARN-4162.v1.001.patch, YARN-4162.v2.001.patch, > YARN-4162.v2.002.patch, YARN-4162.v2.003.patch, YARN-4162.v2.004.patch, > YARN-4162.v2.005.patch, restAndJsonOutput.zip > > > When Node Labels are enabled then REST Scheduler Information should also > provide partition specific queue information similar to the existing Web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4052) Set SO_KEEPALIVE on NM servers
[
https://issues.apache.org/jira/browse/YARN-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961290#comment-14961290
]
Hadoop QA commented on YARN-4052:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 22m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 11m 2s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 14m 10s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 33s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 29s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 59s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 46s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 1s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 27s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 53m 6s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767111/YARN-4052.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 79b8d60 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/9465/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9465/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9465/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9465/console |
This message was automatically generated.
> Set SO_KEEPALIVE on NM servers
> --
>
> Key: YARN-4052
> URL: https://issues.apache.org/jira/browse/YARN-4052
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Nathan Roberts
>Assignee: Chang Li
> Attachments: YARN-4052.2.patch, YARN-4052.patch
>
>
> Shuffle handler does not set SO_KEEPALIVE so we've seen cases where
> FDs/sockets get stuck in ESTABLISHED state indefinitely because the server
> did not see the client leave (network cut or otherwise).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2556) Tool to measure the performance of the timeline server
[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated YARN-2556: --- Attachment: YARN-2556.15.patch Thanks [~sjlee0] for more review! I have fixed the typo in .15 patch. Moreover, I have test the two mapper types(simple entity write mapper and jobhistory files replay mapper) on my single node cluster. Those tests all went well. > Tool to measure the performance of the timeline server > -- > > Key: YARN-2556 > URL: https://issues.apache.org/jira/browse/YARN-2556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Chang Li > Labels: BB2015-05-TBR > Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, > YARN-2556.1.patch, YARN-2556.10.patch, YARN-2556.11.patch, > YARN-2556.12.patch, YARN-2556.13.patch, YARN-2556.13.whitespacefix.patch, > YARN-2556.14.patch, YARN-2556.14.whitespacefix.patch, YARN-2556.15.patch, > YARN-2556.2.patch, YARN-2556.3.patch, YARN-2556.4.patch, YARN-2556.5.patch, > YARN-2556.6.patch, YARN-2556.7.patch, YARN-2556.8.patch, YARN-2556.9.patch, > YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch > > > We need to be able to understand the capacity model for the timeline server > to give users the tools they need to deploy a timeline server with the > correct capacity. > I propose we create a mapreduce job that can measure timeline server write > and read performance. Transactions per second, I/O for both read and write > would be a good start. > This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4179) [reader implementation] support flow activity queries based on time
[ https://issues.apache.org/jira/browse/YARN-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961238#comment-14961238 ] Sangjin Lee commented on YARN-4179: --- bq. Now the question comes should we then have a daterange like "-20151001" or support 2 query params. Will go with former as of now. +1. I think we can stick with a single param, whereas just "20151001" is interpreted to pick only that date. > [reader implementation] support flow activity queries based on time > --- > > Key: YARN-4179 > URL: https://issues.apache.org/jira/browse/YARN-4179 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena >Priority: Minor > Attachments: YARN-4179-YARN-2928.01.patch > > > This came up as part of YARN-4074 and YARN-4075. > Currently the only query pattern that's supported on the flow activity table > is by cluster only. But it might be useful to support queries by cluster and > certain date or dates. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4052) Set SO_KEEPALIVE on NM servers
[ https://issues.apache.org/jira/browse/YARN-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated YARN-4052: --- Attachment: YARN-4052.2.patch .2 patch include a unit test > Set SO_KEEPALIVE on NM servers > -- > > Key: YARN-4052 > URL: https://issues.apache.org/jira/browse/YARN-4052 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: Chang Li > Attachments: YARN-4052.2.patch, YARN-4052.patch > > > Shuffle handler does not set SO_KEEPALIVE so we've seen cases where > FDs/sockets get stuck in ESTABLISHED state indefinitely because the server > did not see the client leave (network cut or otherwise). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961154#comment-14961154 ] Bikas Saha commented on YARN-1509: -- Sounds good > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch, YARN-1509.5.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3216) Max-AM-Resource-Percentage should respect node labels
[ https://issues.apache.org/jira/browse/YARN-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961126#comment-14961126 ] Sunil G commented on YARN-3216: --- Thanks [~leftnoteasy]. I ll also write more tests to see this Cases too. Will upload a patch soon. > Max-AM-Resource-Percentage should respect node labels > - > > Key: YARN-3216 > URL: https://issues.apache.org/jira/browse/YARN-3216 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan >Assignee: Sunil G >Priority: Critical > Attachments: 0001-YARN-3216.patch, 0002-YARN-3216.patch, > 0003-YARN-3216.patch, 0004-YARN-3216.patch, 0005-YARN-3216.patch, > 0006-YARN-3216.patch, 0007-YARN-3216.patch > > > Currently, max-am-resource-percentage considers default_partition only. When > a queue can access multiple partitions, we should be able to compute > max-am-resource-percentage based on that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4243) Add retry on establishing Zookeeper conenction in EmbeddedElectorService#serviceInit
[ https://issues.apache.org/jira/browse/YARN-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961124#comment-14961124 ] Rohith Sharma K S commented on YARN-4243: - I am +1 for the latest patch. The patch also keeps old behavior and gives options for the users to fail fast. > Add retry on establishing Zookeeper conenction in > EmbeddedElectorService#serviceInit > > > Key: YARN-4243 > URL: https://issues.apache.org/jira/browse/YARN-4243 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4243.1.patch, YARN-4243.2.1.patch, > YARN-4243.2.patch, YARN-4243.3.patch, YARN-4243.4.patch > > > Right now, the RM would shut down if the zk connection is down when the RM do > the initialization. We need to add retry on this part -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3216) Max-AM-Resource-Percentage should respect node labels
[ https://issues.apache.org/jira/browse/YARN-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961125#comment-14961125 ] Wangda Tan commented on YARN-3216: -- bq. Yes. For labeled partition, we will only use resource limit per-partition per-queue (based on absolute capacity). For empty label, we will use both. Correct? Yes I think so. > Max-AM-Resource-Percentage should respect node labels > - > > Key: YARN-3216 > URL: https://issues.apache.org/jira/browse/YARN-3216 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan >Assignee: Sunil G >Priority: Critical > Attachments: 0001-YARN-3216.patch, 0002-YARN-3216.patch, > 0003-YARN-3216.patch, 0004-YARN-3216.patch, 0005-YARN-3216.patch, > 0006-YARN-3216.patch, 0007-YARN-3216.patch > > > Currently, max-am-resource-percentage considers default_partition only. When > a queue can access multiple partitions, we should be able to compute > max-am-resource-percentage based on that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3216) Max-AM-Resource-Percentage should respect node labels
[ https://issues.apache.org/jira/browse/YARN-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961123#comment-14961123 ] Sunil G commented on YARN-3216: --- Yes. For labeled partition, we will only use resource limit per-partition per-queue (based on absolute capacity). For empty label, we will use both. Correct? > Max-AM-Resource-Percentage should respect node labels > - > > Key: YARN-3216 > URL: https://issues.apache.org/jira/browse/YARN-3216 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan >Assignee: Sunil G >Priority: Critical > Attachments: 0001-YARN-3216.patch, 0002-YARN-3216.patch, > 0003-YARN-3216.patch, 0004-YARN-3216.patch, 0005-YARN-3216.patch, > 0006-YARN-3216.patch, 0007-YARN-3216.patch > > > Currently, max-am-resource-percentage considers default_partition only. When > a queue can access multiple partitions, we should be able to compute > max-am-resource-percentage based on that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3216) Max-AM-Resource-Percentage should respect node labels
[ https://issues.apache.org/jira/browse/YARN-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961102#comment-14961102 ] Wangda Tan commented on YARN-3216: -- bq. For now we will use queueResourceLimitsInfo.getQueueCurrentLimit() as one limit to calculate am-resource-percent along with configured capacity of partition in queue. I think we should use queueResourceLimitsInfo.getQueueCurrentLimit() as one limit *of non-labeled partition*. > Max-AM-Resource-Percentage should respect node labels > - > > Key: YARN-3216 > URL: https://issues.apache.org/jira/browse/YARN-3216 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan >Assignee: Sunil G >Priority: Critical > Attachments: 0001-YARN-3216.patch, 0002-YARN-3216.patch, > 0003-YARN-3216.patch, 0004-YARN-3216.patch, 0005-YARN-3216.patch, > 0006-YARN-3216.patch, 0007-YARN-3216.patch > > > Currently, max-am-resource-percentage considers default_partition only. When > a queue can access multiple partitions, we should be able to compute > max-am-resource-percentage based on that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961090#comment-14961090
]
Wangda Tan commented on YARN-1509:
--
Sync up with [~mding]. I think we're all on the same page now. Some thoughts
from my side:
- We can separate cancel container resource request change to a separated JIRA.
Now using decrease container to same size to cancel pending increase request is
like a workaround, we'd better add a protocol to AMRMProtocol so AMRMClient can
do things correct.
- For the onContainerResourceChanged API
{code}
public abstract void onContainersResourceChanged(List containers);
{code}
This looks good to me. Since we have a unified API to request change container,
return the updated container to AM should be enough, AM knows if it's an
increased or decreased container.
Thoughts? [~bikassaha].
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3216) Max-AM-Resource-Percentage should respect node labels
[
https://issues.apache.org/jira/browse/YARN-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961077#comment-14961077
]
Sunil G commented on YARN-3216:
---
Sure [~leftnoteasy]. For now we will use
{{queueResourceLimitsInfo.getQueueCurrentLimit()}} as one limit to calculate
am-resource-percent along with configured capacity of partition in queue.
> Max-AM-Resource-Percentage should respect node labels
> -
>
> Key: YARN-3216
> URL: https://issues.apache.org/jira/browse/YARN-3216
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan
>Assignee: Sunil G
>Priority: Critical
> Attachments: 0001-YARN-3216.patch, 0002-YARN-3216.patch,
> 0003-YARN-3216.patch, 0004-YARN-3216.patch, 0005-YARN-3216.patch,
> 0006-YARN-3216.patch, 0007-YARN-3216.patch
>
>
> Currently, max-am-resource-percentage considers default_partition only. When
> a queue can access multiple partitions, we should be able to compute
> max-am-resource-percentage based on that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3216) Max-AM-Resource-Percentage should respect node labels
[ https://issues.apache.org/jira/browse/YARN-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961046#comment-14961046 ] Wangda Tan commented on YARN-3216: -- bq. This can almost match our usecase. and I feel I can file a ticket to handle queue-limit computation for node partition. Thoughts? We need it, but I think the problem is, we don't have an API for application to get headroom for different partitions. IIRC, queue limit is used to compute more precise headroom for application. We can add the queue-limit computation for node partition after we have such API. > Max-AM-Resource-Percentage should respect node labels > - > > Key: YARN-3216 > URL: https://issues.apache.org/jira/browse/YARN-3216 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan >Assignee: Sunil G >Priority: Critical > Attachments: 0001-YARN-3216.patch, 0002-YARN-3216.patch, > 0003-YARN-3216.patch, 0004-YARN-3216.patch, 0005-YARN-3216.patch, > 0006-YARN-3216.patch, 0007-YARN-3216.patch > > > Currently, max-am-resource-percentage considers default_partition only. When > a queue can access multiple partitions, we should be able to compute > max-am-resource-percentage based on that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961035#comment-14961035 ] Bikas Saha commented on YARN-1509: -- Does issuing an increase followed by the decrease actually remove the pending change request on the RM or will it cause the RM to try to change a container resource to the same size as the existing resource and then go down the code path of increase (new token) or decrease (update NM). This would be a corner case that would be good to double check. If that works and the RM actually removes the pending container change request then we could use this mechanism to implement a cancel method wrapper in the AMRMClient. Otherwise, if fixes are needed on the RM side, we could so it separately when we fix the RM. > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch, YARN-1509.5.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4243) Add retry on establishing Zookeeper conenction in EmbeddedElectorService#serviceInit
[ https://issues.apache.org/jira/browse/YARN-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961006#comment-14961006 ] Xuan Gong commented on YARN-4243: - [~kasha] Do we have any other comments ? > Add retry on establishing Zookeeper conenction in > EmbeddedElectorService#serviceInit > > > Key: YARN-4243 > URL: https://issues.apache.org/jira/browse/YARN-4243 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4243.1.patch, YARN-4243.2.1.patch, > YARN-4243.2.patch, YARN-4243.3.patch, YARN-4243.4.patch > > > Right now, the RM would shut down if the zk connection is down when the RM do > the initialization. We need to add retry on this part -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4274) NodeStatusUpdaterImpl should register to RM again after a non-fatal exception happen before
[ https://issues.apache.org/jira/browse/YARN-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-4274: - Description: From YARN-3896, an non-fatal exception like response ID mismatch between NM and RM (due to a race condition) will cause NM stop working. I think we should make it more robust to tolerant a few failure in registering to RM with retry a few times. (was: From YARN-3896, an non-fatal exception like response ID mismatch between NM and RM (due to a race condition) will cause NM stop working. I think we should make it more robust to tolerant a few times failure in registering to RM.) > NodeStatusUpdaterImpl should register to RM again after a non-fatal exception > happen before > --- > > Key: YARN-4274 > URL: https://issues.apache.org/jira/browse/YARN-4274 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Junping Du >Assignee: Junping Du > > From YARN-3896, an non-fatal exception like response ID mismatch between NM > and RM (due to a race condition) will cause NM stop working. I think we > should make it more robust to tolerant a few failure in registering to RM > with retry a few times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3896) RMNode transitioned from RUNNING to REBOOTED because its response id had not been reset synchronously
[
https://issues.apache.org/jira/browse/YARN-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960991#comment-14960991
]
Junping Du commented on YARN-3896:
--
Filed YARN-4274.
> RMNode transitioned from RUNNING to REBOOTED because its response id had not
> been reset synchronously
> -
>
> Key: YARN-3896
> URL: https://issues.apache.org/jira/browse/YARN-3896
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jun Gong
>Assignee: Jun Gong
> Labels: resourcemanager
> Fix For: 2.7.2, 2.6.2
>
> Attachments: 0001-YARN-3896.patch, YARN-3896.01.patch,
> YARN-3896.02.patch, YARN-3896.03.patch, YARN-3896.04.patch,
> YARN-3896.05.patch, YARN-3896.06.patch, YARN-3896.07.patch
>
>
> {noformat}
> 2015-07-03 16:49:39,075 INFO org.apache.hadoop.yarn.util.RackResolver:
> Resolved 10.208.132.153 to /default-rack
> 2015-07-03 16:49:39,075 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
> Reconnect from the node at: 10.208.132.153
> 2015-07-03 16:49:39,075 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
> NodeManager from node 10.208.132.153(cmPort: 8041 httpPort: 8080) registered
> with capability: , assigned nodeId
> 10.208.132.153:8041
> 2015-07-03 16:49:39,104 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Too far
> behind rm response id:2506413 nm response id:0
> 2015-07-03 16:49:39,137 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Deactivating
> Node 10.208.132.153:8041 as it is now REBOOTED
> 2015-07-03 16:49:39,137 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> 10.208.132.153:8041 Node Transitioned from RUNNING to REBOOTED
> {noformat}
> The node(10.208.132.153) reconnected with RM. When it registered with RM, RM
> set its lastNodeHeartbeatResponse's id to 0 asynchronously. But the node's
> heartbeat come before RM succeeded setting the id to 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4274) NodeStatusUpdaterImpl should register to RM again after a non-fatal exception happen before
Junping Du created YARN-4274: Summary: NodeStatusUpdaterImpl should register to RM again after a non-fatal exception happen before Key: YARN-4274 URL: https://issues.apache.org/jira/browse/YARN-4274 Project: Hadoop YARN Issue Type: Improvement Reporter: Junping Du Assignee: Junping Du >From YARN-3896, an non-fatal exception like response ID mismatch between NM >and RM (due to a race condition) will cause NM stop working. I think we should >make it more robust to tolerant a few times failure in registering to RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3896) RMNode transitioned from RUNNING to REBOOTED because its response id had not been reset synchronously
[
https://issues.apache.org/jira/browse/YARN-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960972#comment-14960972
]
Junping Du commented on YARN-3896:
--
I think we may also need to fix
NodeStatusUpdaterImpl.rebootNodeStatusUpdaterAndRegisterWithRM() to retry the
register to RM when have non-fatal exception. Will file a separate JIRA to
fix/discuss this.
> RMNode transitioned from RUNNING to REBOOTED because its response id had not
> been reset synchronously
> -
>
> Key: YARN-3896
> URL: https://issues.apache.org/jira/browse/YARN-3896
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Jun Gong
>Assignee: Jun Gong
> Labels: resourcemanager
> Fix For: 2.7.2, 2.6.2
>
> Attachments: 0001-YARN-3896.patch, YARN-3896.01.patch,
> YARN-3896.02.patch, YARN-3896.03.patch, YARN-3896.04.patch,
> YARN-3896.05.patch, YARN-3896.06.patch, YARN-3896.07.patch
>
>
> {noformat}
> 2015-07-03 16:49:39,075 INFO org.apache.hadoop.yarn.util.RackResolver:
> Resolved 10.208.132.153 to /default-rack
> 2015-07-03 16:49:39,075 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
> Reconnect from the node at: 10.208.132.153
> 2015-07-03 16:49:39,075 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
> NodeManager from node 10.208.132.153(cmPort: 8041 httpPort: 8080) registered
> with capability: , assigned nodeId
> 10.208.132.153:8041
> 2015-07-03 16:49:39,104 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Too far
> behind rm response id:2506413 nm response id:0
> 2015-07-03 16:49:39,137 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Deactivating
> Node 10.208.132.153:8041 as it is now REBOOTED
> 2015-07-03 16:49:39,137 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> 10.208.132.153:8041 Node Transitioned from RUNNING to REBOOTED
> {noformat}
> The node(10.208.132.153) reconnected with RM. When it registered with RM, RM
> set its lastNodeHeartbeatResponse's id to 0 asynchronously. But the node's
> heartbeat come before RM succeeded setting the id to 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3448) Add Rolling Time To Lives Level DB Plugin Capabilities
[ https://issues.apache.org/jira/browse/YARN-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960934#comment-14960934 ] Jonathan Eagles commented on YARN-3448: --- The fromTs implementation does not work at scale due to design limitations. All matching entity types are stored in sorted order by start time. In order to ask the question fromTs you must do a full db scan to check matching entries. Entries would have to be sorted by insert time to make this query work efficiently. Throwing an exception that this functionality isn't support makes a lot of sense. > Add Rolling Time To Lives Level DB Plugin Capabilities > -- > > Key: YARN-3448 > URL: https://issues.apache.org/jira/browse/YARN-3448 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 2.8.0 > > Attachments: YARN-3448.1.patch, YARN-3448.10.patch, > YARN-3448.12.patch, YARN-3448.13.patch, YARN-3448.14.patch, > YARN-3448.15.patch, YARN-3448.16.patch, YARN-3448.17.patch, > YARN-3448.2.patch, YARN-3448.3.patch, YARN-3448.4.patch, YARN-3448.5.patch, > YARN-3448.7.patch, YARN-3448.8.patch, YARN-3448.9.patch > > > For large applications, the majority of the time in LeveldbTimelineStore is > spent deleting old entities record at a time. An exclusive write lock is held > during the entire deletion phase which in practice can be hours. If we are to > relax some of the consistency constraints, other performance enhancing > techniques can be employed to maximize the throughput and minimize locking > time. > Split the 5 sections of the leveldb database (domain, owner, start time, > entity, index) into 5 separate databases. This allows each database to > maximize the read cache effectiveness based on the unique usage patterns of > each database. With 5 separate databases each lookup is much faster. This can > also help with I/O to have the entity and index databases on separate disks. > Rolling DBs for entity and index DBs. 99.9% of the data are in these two > sections 4:1 ration (index to entity) at least for tez. We replace DB record > removal with file system removal if we create a rolling set of databases that > age out and can be efficiently removed. To do this we must place a constraint > to always place an entity's events into it's correct rolling db instance > based on start time. This allows us to stitching the data back together while > reading and artificial paging. > Relax the synchronous writes constraints. If we are willing to accept losing > some records that we not flushed in the operating system during a crash, we > can use async writes that can be much faster. > Prefer Sequential writes. sequential writes can be several times faster than > random writes. Spend some small effort arranging the writes in such a way > that will trend towards sequential write performance over random write > performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960927#comment-14960927
]
MENG DING commented on YARN-1509:
-
Hi, [~bikassaha]
Thanks for the comments.
I probably didn't make myself clear. We are on the SAME page that for the sake
of point 2 alone, it already makes sense to combine increase/decrease API into
one change API: {code}public abstract void
requestContainerResourceChange(Container container, Resource capability);{code}
What I was trying to say in the previous post is that to support mix of
increase and decrease in one change request (point 1) doesn't seem to be very
feasible (even at a later date). But I don't think we need to worry about that
for now.
Since we are combining increase/decrease API, we definitely should combine the
callback methods into one as well: onContainerResourceChanged(). At this point,
I am inclined to simply do the following which doesn't incur much code changes.
I will discuss further with [~leftnoteasy] on this. {code}public abstract void
onContainersResourceChanged(List containers);{code}
bq. Or are you saying that invalid container resource change requests are
immediately rejected by the RM synchronously in the allocate RPC?
Yes, the ApplicationMasterService will perform a series of sanity checks (e.g.,
requested resource <= maximum allocation, etc), and reject invalid requests
immediately. This is the same for other requests too.
bq. Having a simple cancel request regardless of increase or decrease is
preferable since then we are not leaking the current state of the
implementation to the user. It is future safe
Make sense to me. We can probably have something like
{{cancelContainerResourceChange(Container container)}}, which applies to the
container that has an outstanding pending increase sitting in the pendingChange
map. There is no explicit protocol to support cancellation of resource change
yet. For now we can achieve that by issuing a backend decrease request with the
target resource set to the same as the current resource, which effectively
cancels any outstanding increase request.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4273) Containers can be leaked due to race between application being killed and NM registering back after recovery
[ https://issues.apache.org/jira/browse/YARN-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960791#comment-14960791 ] Jason Lowe commented on YARN-4273: -- Wait, isn't the RM supposed to be remembering to tell the NM about the application finishing? When the NM registers it not only will say which containers are still running but also which applications are still active. When the application finishes (due to being killed or whatever) then the RM will tell the NM that the app is finished, either directly on the reconnect event because the app is unknown or from RMAppImpl when it gets the APP_RUNNING_ON_NODE event. So I don't think these will be leaked forever. Eventually the app will finish and the NM will be told about it. When the NM hears the app is finished, it will kill all containers belonging to that app. > Containers can be leaked due to race between application being killed and NM > registering back after recovery > > > Key: YARN-4273 > URL: https://issues.apache.org/jira/browse/YARN-4273 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.1 >Reporter: Varun Saxena >Assignee: Varun Saxena > > This issue is based on discussion on YARN-4000 > Consider this scenario : > 1) Application is recovered and added into scheduler, some slow NM has not > re-registered back, so those containers are not yet recovered. > 2) User kills this app > 3) CapacityScheduler#doneApplicationAttempt is called, containers tracked by > RM so far are killed. Note that CapacityScheduler#doneApplication is not > called, so scheduler still has the SchedulerApplication in memory > 4) Slow NM now re-registers and try to recover the containers. If application > is set to keep containers across attempts, these containers will be recovered > even though application is in the process of being killed. These container > will not be killed later on. Hence, these containers are leaked. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4273) Containers can be leaked due to race between application being killed and NM registering back after recovery
Varun Saxena created YARN-4273: -- Summary: Containers can be leaked due to race between application being killed and NM registering back after recovery Key: YARN-4273 URL: https://issues.apache.org/jira/browse/YARN-4273 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.1 Reporter: Varun Saxena Assignee: Varun Saxena This issue is based on discussion on YARN-4000 Consider this scenario : 1) Application is recovered and added into scheduler, some slow NM has not re-registered back, so those containers are not yet recovered. 2) User kills this app 3) CapacityScheduler#doneApplicationAttempt is called, containers tracked by RM so far are killed. Note that CapacityScheduler#doneApplication is not called, so scheduler still has the SchedulerApplication in memory 4) Slow NM now re-registers and try to recover the containers. If application is set to keep containers across attempts, these containers will be recovered even though application is in the process of being killed. These container will not be killed later on. Hence, these containers are leaked. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (YARN-4262) Allow whitelisted users to run privileged docker containers.
[ https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960505#comment-14960505 ] Varun Vasudev edited comment on YARN-4262 at 10/16/15 10:50 AM: +1. I'll commit this on Monday if no one objects. was (Author: vvasudev): +1. I'll commit this on Mnonday if no one objects. > Allow whitelisted users to run privileged docker containers. > - > > Key: YARN-4262 > URL: https://issues.apache.org/jira/browse/YARN-4262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4262.001.patch, YARN-4262.002.patch, > YARN-4262.003.patch > > > (Updated based on discussion in the JIRA) > There are scenarios where privileged containers are necessary in order to run > certain kinds of applications (one example is trying to run postresql/oracle > inside containers). However, given the security implications, we should > ensure that : > 1) privileged containers are disabled by default > 2) if enabled, only a whitelisted set of users should be allowed to launch > such containers and > 3) Not all containers launched by whitelisted users need to be privileged > containers : whitelisted users need to explicitly request that a privileged > container be launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4262) Allow whitelisted users to run privileged docker containers.
[ https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960505#comment-14960505 ] Varun Vasudev commented on YARN-4262: - +1. I'll commit this on Mnonday if no one objects. > Allow whitelisted users to run privileged docker containers. > - > > Key: YARN-4262 > URL: https://issues.apache.org/jira/browse/YARN-4262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4262.001.patch, YARN-4262.002.patch, > YARN-4262.003.patch > > > (Updated based on discussion in the JIRA) > There are scenarios where privileged containers are necessary in order to run > certain kinds of applications (one example is trying to run postresql/oracle > inside containers). However, given the security implications, we should > ensure that : > 1) privileged containers are disabled by default > 2) if enabled, only a whitelisted set of users should be allowed to launch > such containers and > 3) Not all containers launched by whitelisted users need to be privileged > containers : whitelisted users need to explicitly request that a privileged > container be launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4272) CLI MiniCluster fails to launch MiniYARNCluster due to NoClassDefFoundError
[ https://issues.apache.org/jira/browse/YARN-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-4272: --- Description: [CLI MiniCluster|https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/CLIMiniCluster.html] fails due to NoClassDefFoundError because the hadoop-yarn-server-tests-*-SNAPSHOT-tests.jar containing MiniYARNCluster is not included in the classpath. (was: [CLI MiniCluster|https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/CLIMiniCluster.html] fails due to NoClassDefFoundError because the hadoop-yarn-server-tests-*-SNAPSHOT-tests.jar containing MiniYARNCluster is not provided by dist package.) > CLI MiniCluster fails to launch MiniYARNCluster due to NoClassDefFoundError > --- > > Key: YARN-4272 > URL: https://issues.apache.org/jira/browse/YARN-4272 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > > [CLI > MiniCluster|https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/CLIMiniCluster.html] > fails due to NoClassDefFoundError because the > hadoop-yarn-server-tests-*-SNAPSHOT-tests.jar containing MiniYARNCluster is > not included in the classpath. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4068) Support appUpdated event in TimelineV2 to publish details for movetoqueue, change in priority
[ https://issues.apache.org/jira/browse/YARN-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960364#comment-14960364 ] Naganarasimha G R commented on YARN-4068: - Hi [~sunilg] During the latest branch rebase, [~sjlee0] as already replicated the patch in ATS v2. so i think this jira can be closed. Can you please verify and confirm > Support appUpdated event in TimelineV2 to publish details for movetoqueue, > change in priority > - > > Key: YARN-4068 > URL: https://issues.apache.org/jira/browse/YARN-4068 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sunil G >Assignee: Sunil G > > YARN-4044 supports appUpdated event changes to TimelineV1. This jira is to > track and port appUpdated changes in V2 for > - movetoqueue > - updateAppPriority -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3798) ZKRMStateStore shouldn't create new session without occurrance of SESSIONEXPIED
[
https://issues.apache.org/jira/browse/YARN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960357#comment-14960357
]
Tsuyoshi Ozawa commented on YARN-3798:
--
Oh, I think the review is done by [~zxu]. It's ready for review. I'm checking
this in.
> ZKRMStateStore shouldn't create new session without occurrance of
> SESSIONEXPIED
> ---
>
> Key: YARN-3798
> URL: https://issues.apache.org/jira/browse/YARN-3798
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.0
> Environment: Suse 11 Sp3
>Reporter: Bibin A Chundatt
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: RM.log, YARN-3798-2.7.002.patch,
> YARN-3798-branch-2.6.01.patch, YARN-3798-branch-2.7.002.patch,
> YARN-3798-branch-2.7.003.patch, YARN-3798-branch-2.7.004.patch,
> YARN-3798-branch-2.7.005.patch, YARN-3798-branch-2.7.006.patch,
> YARN-3798-branch-2.7.patch
>
>
> RM going down with NoNode exception during create of znode for appattempt
> *Please find the exception logs*
> {code}
> 2015-06-09 10:09:44,732 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore:
> ZKRMStateStore Session connected
> 2015-06-09 10:09:44,732 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore:
> ZKRMStateStore Session restored
> 2015-06-09 10:09:44,886 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore:
> Exception while executing a ZK operation.
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1405)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1310)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:926)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:923)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1122)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:923)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:937)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createWithRetries(ZKRMStateStore.java:970)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.updateApplicationAttemptStateInternal(ZKRMStateStore.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:260)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:837)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:900)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:895)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
> at java.lang.Thread.run(Thread.java:745)
> 2015-06-09 10:09:44,887 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore: Maxed
> out ZK retries. Giving up!
> 2015-06-09 10:09:44,887 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Error
> updating appAttempt: appattempt_1433764310492_7152_01
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1405)
>
[jira] [Commented] (YARN-3798) ZKRMStateStore shouldn't create new session without occurrance of SESSIONEXPIED
[
https://issues.apache.org/jira/browse/YARN-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960352#comment-14960352
]
Tsuyoshi Ozawa commented on YARN-3798:
--
[~sjlee0] thank you for pinging. Tests pass locally on both branch-2.6 and
branch-2.7. Currently, it's waiting for review.
> ZKRMStateStore shouldn't create new session without occurrance of
> SESSIONEXPIED
> ---
>
> Key: YARN-3798
> URL: https://issues.apache.org/jira/browse/YARN-3798
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.0
> Environment: Suse 11 Sp3
>Reporter: Bibin A Chundatt
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: RM.log, YARN-3798-2.7.002.patch,
> YARN-3798-branch-2.6.01.patch, YARN-3798-branch-2.7.002.patch,
> YARN-3798-branch-2.7.003.patch, YARN-3798-branch-2.7.004.patch,
> YARN-3798-branch-2.7.005.patch, YARN-3798-branch-2.7.006.patch,
> YARN-3798-branch-2.7.patch
>
>
> RM going down with NoNode exception during create of znode for appattempt
> *Please find the exception logs*
> {code}
> 2015-06-09 10:09:44,732 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore:
> ZKRMStateStore Session connected
> 2015-06-09 10:09:44,732 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore:
> ZKRMStateStore Session restored
> 2015-06-09 10:09:44,886 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore:
> Exception while executing a ZK operation.
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1405)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1310)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:926)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:923)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1122)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:923)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:937)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createWithRetries(ZKRMStateStore.java:970)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.updateApplicationAttemptStateInternal(ZKRMStateStore.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:260)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:837)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:900)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:895)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
> at java.lang.Thread.run(Thread.java:745)
> 2015-06-09 10:09:44,887 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore: Maxed
> out ZK retries. Giving up!
> 2015-06-09 10:09:44,887 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Error
> updating appAttempt: appattempt_1433764310492_7152_01
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
> at org.apache.zookeeper.ZooKeeper.m
[jira] [Commented] (YARN-4262) Allow whitelisted users to run privileged docker containers.
[
https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960337#comment-14960337
]
Hadoop QA commented on YARN-4262:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 52s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 49s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 16s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 49s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 32s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 22s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 2m 6s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 8m 46s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 58m 46s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767002/YARN-4262.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / cf23f2c |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9464/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9464/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9464/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9464/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9464/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9464/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9464/console |
This message was automatically generated.
> Allow whitelisted users to run privileged docker containers.
> -
>
> Key: YARN-4262
> URL: https://issues.apache.org/jira/browse/YARN-4262
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
> Attachments: YARN-4262.001.patch, YARN-4262.002.patch,
> YARN-4262.003.patch
>
>
> (Updated based on discussion in the JIRA)
> There are scenarios where privileged containers are necessary in order to run
> certain kinds of applications (one example is trying to run postresql/oracle
> inside containers). However, given the security implications, we should
> ensure that :
> 1) privileged containers are disabled by default
> 2) if enabled, only a whitelisted set of users should be allowed to launch
> such containers and
> 3) Not all containers launched by whitelisted users need to be privileged
> containers : whitelisted users need to explicitly request that a privileged
> container be launched.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4272) CLI MiniCluster fails to launch MiniYARNCluster due to NoClassDefFoundError
[ https://issues.apache.org/jira/browse/YARN-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-4272: --- Description: [CLI MiniCluster|https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/CLIMiniCluster.html] fails due to NoClassDefFoundError because the hadoop-yarn-server-tests-*-SNAPSHOT-tests.jar containing MiniYARNCluster is not provided by dist package. (was: [CLI MiniCluster|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CLIMiniCluster.html] fails due to NoClassDefFoundError because the hadoop-yarn-server-tests-*-SNAPSHOT-tests.jar containing MiniYARNCluster is not provided by dist package.) > CLI MiniCluster fails to launch MiniYARNCluster due to NoClassDefFoundError > --- > > Key: YARN-4272 > URL: https://issues.apache.org/jira/browse/YARN-4272 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > > [CLI > MiniCluster|https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/CLIMiniCluster.html] > fails due to NoClassDefFoundError because the > hadoop-yarn-server-tests-*-SNAPSHOT-tests.jar containing MiniYARNCluster is > not provided by dist package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4272) CLI MiniCluster fails to launch MiniYARNCluster due to NoClassDefFoundError
[
https://issues.apache.org/jira/browse/YARN-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960318#comment-14960318
]
Masatake Iwasaki commented on YARN-4272:
{noformat}
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/server/MiniYARNCluster
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:170)
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:129)
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:314)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at
org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:130)
at
org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.yarn.server.MiniYARNCluster
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.Class
{noformat}
> CLI MiniCluster fails to launch MiniYARNCluster due to NoClassDefFoundError
> ---
>
> Key: YARN-4272
> URL: https://issues.apache.org/jira/browse/YARN-4272
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>
> [CLI
> MiniCluster|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CLIMiniCluster.html]
> fails due to NoClassDefFoundError because the
> hadoop-yarn-server-tests-*-SNAPSHOT-tests.jar containing MiniYARNCluster is
> not provided by dist package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4272) CLI MiniCluster fails to launch MiniYARNCluster due to NoClassDefFoundError
[ https://issues.apache.org/jira/browse/YARN-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960307#comment-14960307 ] Masatake Iwasaki commented on YARN-4272: CLI MiniCluster is not affected by HDFS-9226 because mockito is provided by dist package (under share/hadoop/common/lib). > CLI MiniCluster fails to launch MiniYARNCluster due to NoClassDefFoundError > --- > > Key: YARN-4272 > URL: https://issues.apache.org/jira/browse/YARN-4272 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > > [CLI > MiniCluster|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CLIMiniCluster.html] > fails due to NoClassDefFoundError because the > hadoop-yarn-server-tests-*-SNAPSHOT-tests.jar containing MiniYARNCluster is > not provided by dist package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4272) CLI MiniCluster fails to launch MiniYARNCluster due to NoClassDefFoundError
Masatake Iwasaki created YARN-4272: -- Summary: CLI MiniCluster fails to launch MiniYARNCluster due to NoClassDefFoundError Key: YARN-4272 URL: https://issues.apache.org/jira/browse/YARN-4272 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.1 Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki [CLI MiniCluster|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CLIMiniCluster.html] fails due to NoClassDefFoundError because the hadoop-yarn-server-tests-*-SNAPSHOT-tests.jar containing MiniYARNCluster is not provided by dist package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4170) AM need to be notified with priority in AllocateResponse
[
https://issues.apache.org/jira/browse/YARN-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960298#comment-14960298
]
Hadoop QA commented on YARN-4170:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 8s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 11s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 1s | The applied patch generated 1
new checkstyle issues (total was 7, now 8). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 4m 40s | The patch appears to introduce 3
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 2m 0s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 57m 31s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 107m 56s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-common |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766553/0007-YARN-4170.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / cf23f2c |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9463/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9463/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9463/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9463/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9463/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9463/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9463/console |
This message was automatically generated.
> AM need to be notified with priority in AllocateResponse
> -
>
> Key: YARN-4170
> URL: https://issues.apache.org/jira/browse/YARN-4170
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-4170.patch, 0002-YARN-4170.patch,
> 0003-YARN-4170.patch, 0004-YARN-4170.patch, 0005-YARN-4170-v2.patch,
> 0006-YARN-4170.patch, 0007-YARN-4170.patch
>
>
> As discussed in MAPREDUCE-5870, Application Master need to be notified with
> priority in Allocate heartbeat. This will help AM to know the priority and
> can update JobStatus when client asks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4267) Add additional logging to container launch implementations in container-executor
[ https://issues.apache.org/jira/browse/YARN-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960276#comment-14960276 ] Varun Vasudev commented on YARN-4267: - +1. I'll commit this on Monday if no one objects. > Add additional logging to container launch implementations in > container-executor > > > Key: YARN-4267 > URL: https://issues.apache.org/jira/browse/YARN-4267 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4267.001.patch > > > The launch container implementations in container-executor involve several > steps and when a container launch fails, it is not always evident what steps > were successfully completed. This is particularly true of launching docker > containers. Additional logging would help in diagnosing problems/debug > issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4262) Allow whitelisted users to run privileged docker containers.
[ https://issues.apache.org/jira/browse/YARN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidharta Seethana updated YARN-4262: Attachment: YARN-4262.003.patch Thanks, [~vvasudev]. Uploaded a new patch with the suggested fixes in place. > Allow whitelisted users to run privileged docker containers. > - > > Key: YARN-4262 > URL: https://issues.apache.org/jira/browse/YARN-4262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4262.001.patch, YARN-4262.002.patch, > YARN-4262.003.patch > > > (Updated based on discussion in the JIRA) > There are scenarios where privileged containers are necessary in order to run > certain kinds of applications (one example is trying to run postresql/oracle > inside containers). However, given the security implications, we should > ensure that : > 1) privileged containers are disabled by default > 2) if enabled, only a whitelisted set of users should be allowed to launch > such containers and > 3) Not all containers launched by whitelisted users need to be privileged > containers : whitelisted users need to explicitly request that a privileged > container be launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
