[jira] [Created] (YARN-5984) Refactor move application across queue's CS level implementation

[Sunil G (JIRA)]

Sunil G created YARN-5984:
-

 Summary: Refactor move application across queue's CS level 
implementation
 Key: YARN-5984
 URL: https://issues.apache.org/jira/browse/YARN-5984
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacity scheduler, resourcemanager
Reporter: Sunil G
Assignee: Sunil G


Currently we use a top level write lock in CS#moveApplication. Also we are 
using few submission time apis in move. This jira will be focussing on coming 
up with a cleaner implementation for moveApplication and could try to share 
code with FS as possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5983) [Umbrella] Support for FPGA as a Resource in YARN

[Zhankun Tang (JIRA)]

Zhankun Tang created YARN-5983:
--

 Summary: [Umbrella] Support for FPGA as a Resource in YARN
 Key: YARN-5983
 URL: https://issues.apache.org/jira/browse/YARN-5983
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: yarn
Reporter: Zhankun Tang
Assignee: Zhankun Tang


As various big data workload running on YARN, CPU no longer scale eventually 
and heterogeneous systems become more important. And ML/DL is a rising star in 
recent years, applications focused on these areas have to utilize GPU or FPGA 
to boost performance. Also, hardware vendors such as Intel also invest in such 
hardware. It is most likely that FPGA will become popular in data centers like 
CPU in the near future.

So YARN as a resource manager should evolve to support this. This JIRA propose 
FPGA to be first-class citizen. The changes roughly includes:
1. FPGA resource detection and heartbeat
2. Scheduler changes
3. FPGA related preparation and isolation before launch container
We know that YARN-3926 is trying to extend current resource model. But still we 
can leave some FPGA related discussion here



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5982) Simplifying some opportunistic container parameters and metrics

[Konstantinos Karanasos (JIRA)]

Konstantinos Karanasos created YARN-5982:


 Summary: Simplifying some opportunistic container parameters and 
metrics
 Key: YARN-5982
 URL: https://issues.apache.org/jira/browse/YARN-5982
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Konstantinos Karanasos
Assignee: Konstantinos Karanasos


This JIRA removes some of the parameters that are related to opportunistic 
containers (e.g., min/max memory/cpu). Instead, we will be using the parameters 
already used by guaranteed containers.
The goal is to reduce the number of parameters that need to be used by the user.

We also fix a small issue related to the container metrics (opportunistic 
memory reported in GB in Web UI, although it was in MB).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Continued] [Release thread] 2.8.0 release activities

[Junping Du]

Thanks Akira for reporting this. Actually, HADOOP-2.8-JACC worked well for 
several runs before last weekend, but it get failed for latest several runs, 
probably affected by recently Jenkins down. 
However, from my recent manual kick off, it seems to be good again: 
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/9/. I will 
keep an eye for today's nightly run to see if it back to normal.


Thanks,

Junping


From: Akira Ajisaka 
Sent: Wednesday, December 07, 2016 12:12 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks Junping and Andrew!

HADOOP-2.8-JACC is not working well, so I manually kicked a job to
compare 2.8 with 2.7.3.

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/24/artifact/target/compat-check/report.html

Regards,
Akira

On 2016/12/02 2:08, Junping Du wrote:
> Thanks Andrew! That's also a nice suggestion. I already create a similar job: 
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/ for 2.8 
> and kick off several runs manually. Will monitor incompatible status from 
> there.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Andrew Wang 
> Sent: Wednesday, November 30, 2016 4:18 PM
> To: Junping Du
> Cc: Sangjin Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
> mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Vinod 
> Vavilapalli; Jian He; Wangda Tan; sj...@twitter.com
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> I recommend giving the JACC report another look. I set up a parameterized 
> jenkins job which you can trigger manually for branch-2.8:
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/
>
> On Wed, Nov 30, 2016 at 4:06 PM, Junping Du 
> > wrote:
> Hi Sangjin and all,
>
>  That sounds good. If anyone have priority fix to backport , please 
> nominate it by following this email thread or ping me on JIRA directly. And I 
> agree that we should keep in mind that our bar for 2.8 release should be high 
> now as we want to move quickly on this release. Non-critical fixes can wait 
> for next one.
>
>   Any other comments and suggestions?
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: sjl...@gmail.com 
> > on behalf of Sangjin Lee 
> >
> Sent: Wednesday, November 30, 2016 3:24 PM
> To: Junping Du
> Cc: common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; 
> mapreduce-...@hadoop.apache.org; 
> yarn-dev@hadoop.apache.org; Vinod 
> Vavilapalli; Jian He; Wangda Tan; sj...@twitter.com
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> Thanks for picking this up Junping!
>
> I heard on email threads and offline discussions that there may be interests 
> in porting some fixes from branch-2 to branch-2.8. I still feel that the bar 
> should be pretty high to do that, but it might be good to have folks suggest 
> some critical fixes that need to be released sooner. Thoughts?
>
> On Tue, Nov 29, 2016 at 8:50 PM, Junping Du 
> >>
>  wrote:
> Hi folks,
>  Per asked by Vinod offline, I would like to continue the effort to push 
> 2.8 release out in next several weeks. The plan is as following:
>  - We only have 5 blockers and 3 critical issues 
> (https://s.apache.org/6kwx) so far, and most of them are in good progress 
> now. I put them all on apache ciwiki ( 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release) and 
> will do daily monitor and update on these issues in next couple of weeks.
>  - When all blocker/critical issues are gone (target EOD is Dec. 10), I 
> will push out all major/minor issues that is open. So if you are working on 
> some major issues and target for release in 2.8, please keep track of our 
> release status and your work progress.
>  - I will create RC immediately once our 2.8 target issues are clean 
> (especially for blocker and critical issues) for voting process.
>
>  I hope we can make it within year 2016, best before Xmas holiday. And I 
> need help from all of you. :)
>
>
> Thanks,
>
> Junping
>
> 
> From: Vinod Kumar Vavilapalli 
> >>
> Sent: Monday, July 25, 2016 2:57 PM
> To: Jian He
> Cc: 

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/179/

[Dec 6, 2016 2:06:50 PM] (kihwal) HDFS-11198. NN UI should link DN web address 
using hostnames.
[Dec 6, 2016 5:00:43 PM] (brahma) HDFS-11178. 
TestAddStripedBlockInFBR#testAddBlockInFullBlockReport fails
[Dec 6, 2016 5:12:11 PM] (liuml07) HDFS-11211. Add a time unit to the DataNode 
client trace format.
[Dec 6, 2016 7:05:47 PM] (xyao) HDFS-10930. Refactor: Wrap Datanode IO related 
operations. Contributed
[Dec 6, 2016 8:04:04 PM] (xiao) HADOOP-13827. Add reencryptEncryptedKey 
interface to KMS.
[Dec 6, 2016 8:49:38 PM] (wang) HDFS-11172. Support an erasure coding policy 
using RS 10 + 4.
[Dec 6, 2016 9:11:03 PM] (rkanter) MAPREDUCE-6571. JobEndNotification info logs 
are missing in AM container
[Dec 6, 2016 9:41:08 PM] (aajisaka) MAPREDUCE-6816. Progress bars in Web UI 
always at 100%. Contributed by
[Dec 6, 2016 10:10:58 PM] (kasha) HADOOP-13859. TestConfigurationFieldsBase 
fails for fields that are
[Dec 6, 2016 10:36:41 PM] (junping_du) YARN-5184. Fix up incompatible changes 
introduced on ContainerStatus and
[Dec 7, 2016 5:09:14 AM] (rohithsharmaks) YARN-5932. Retrospect 
moveApplicationToQueue in align with YARN-5611.




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker 
   hadoop.hdfs.server.datanode.TestBPOfferService 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService 
   hadoop.yarn.server.nodemanager.TestNodeManagerShutdown 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.timelineservice.storage.common.TestRowKeys 
   hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters 
   hadoop.yarn.server.timelineservice.storage.common.TestSeparator 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   hadoop.yarn.server.resourcemanager.TestResourceTrackerService 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
 
   
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities 
   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun 
   
hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl
 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/179/artifact/out/patch-compile-root.txt
  [172K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/179/artifact/out/patch-compile-root.txt
  [172K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/179/artifact/out/patch-compile-root.txt
  [172K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/179/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [200K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/179/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/179/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [52K]
   

[jira] [Created] (YARN-5981) Coprocessor related code changes/cleanup pending HBASE-17273

[Vrushali C (JIRA)]

Vrushali C created YARN-5981:


 Summary: Coprocessor related code changes/cleanup pending 
HBASE-17273
 Key: YARN-5981
 URL: https://issues.apache.org/jira/browse/YARN-5981
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C



Per HBASE-17273, we are looking into if/how the coprocessor code can be moved 
out from yarn/timelineservice into hbase/coprocessor itself. 

If/when this is done, the timeline service code will need to be updated to 
remove the actual coprocessor code, it's references as well as the 
documentation to reflect how to use this new thing. 

Version of hbase used would likely also be changing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5979) Make ApplicationReport @Evolving

[Akira Ajisaka (JIRA)]

Akira Ajisaka created YARN-5979:
---

 Summary: Make ApplicationReport @Evolving
 Key: YARN-5979
 URL: https://issues.apache.org/jira/browse/YARN-5979
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api
Reporter: Akira Ajisaka


Abstract class ApplicationReport is {{@Public}} and {{@Stable}}, but some 
methods are added between minor releases and this breaks source-compatibility. 
We should make it {{@Evolving}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Continued] [Release thread] 2.8.0 release activities

[Akira Ajisaka]

Thanks Junping and Andrew!

HADOOP-2.8-JACC is not working well, so I manually kicked a job to 
compare 2.8 with 2.7.3.


https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/24/artifact/target/compat-check/report.html

Regards,
Akira

On 2016/12/02 2:08, Junping Du wrote:

Thanks Andrew! That's also a nice suggestion. I already create a similar job: 
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/ for 2.8 and 
kick off several runs manually. Will monitor incompatible status from there.


Thanks,


Junping



From: Andrew Wang 
Sent: Wednesday, November 30, 2016 4:18 PM
To: Junping Du
Cc: Sangjin Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Vinod Vavilapalli; 
Jian He; Wangda Tan; sj...@twitter.com
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

I recommend giving the JACC report another look. I set up a parameterized 
jenkins job which you can trigger manually for branch-2.8:

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/

On Wed, Nov 30, 2016 at 4:06 PM, Junping Du 
> wrote:
Hi Sangjin and all,

 That sounds good. If anyone have priority fix to backport , please 
nominate it by following this email thread or ping me on JIRA directly. And I 
agree that we should keep in mind that our bar for 2.8 release should be high 
now as we want to move quickly on this release. Non-critical fixes can wait for 
next one.

  Any other comments and suggestions?


Thanks,


Junping



From: sjl...@gmail.com > 
on behalf of Sangjin Lee >
Sent: Wednesday, November 30, 2016 3:24 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; Vinod Vavilapalli; Jian He; Wangda Tan; 
sj...@twitter.com
Subject: Re: [Continued] [Release thread] 2.8.0 release activities

Thanks for picking this up Junping!

I heard on email threads and offline discussions that there may be interests in 
porting some fixes from branch-2 to branch-2.8. I still feel that the bar 
should be pretty high to do that, but it might be good to have folks suggest 
some critical fixes that need to be released sooner. Thoughts?

On Tue, Nov 29, 2016 at 8:50 PM, Junping Du 
>>
 wrote:
Hi folks,
 Per asked by Vinod offline, I would like to continue the effort to push 
2.8 release out in next several weeks. The plan is as following:
 - We only have 5 blockers and 3 critical issues 
(https://s.apache.org/6kwx) so far, and most of them are in good progress now. 
I put them all on apache ciwiki ( 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release) and will 
do daily monitor and update on these issues in next couple of weeks.
 - When all blocker/critical issues are gone (target EOD is Dec. 10), I 
will push out all major/minor issues that is open. So if you are working on 
some major issues and target for release in 2.8, please keep track of our 
release status and your work progress.
 - I will create RC immediately once our 2.8 target issues are clean 
(especially for blocker and critical issues) for voting process.

 I hope we can make it within year 2016, best before Xmas holiday. And I 
need help from all of you. :)


Thanks,

Junping


From: Vinod Kumar Vavilapalli 
>>
Sent: Monday, July 25, 2016 2:57 PM
To: Jian He
Cc: 
mapreduce-...@hadoop.apache.org>;
 yarn-dev@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org>; 
common-...@hadoop.apache.org>
Subject: Re: [Release thread] 2.8.0 release activities

Thanks for all that great help moving this release forward, Jian, Sangjin, 
Wangda et al! Appreciate it.

The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0 last week.

I only see 9 blocker / critical fixes pending for 2.8.0 -