Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Xiao Chen]

+1 (binding)

- downloaded src tarball, verified md5
- built from source with jdk1.8.0_112
- started a pseudo cluster with hdfs and kms
- sanity checked encryption related operations working
- sanity checked webui and logs.

-Xiao

On Mon, Dec 11, 2017 at 6:10 PM, Aaron T. Myers  wrote:

> +1 (binding)
>
> - downloaded the src tarball and built the source (-Pdist -Pnative)
> - verified the checksum
> - brought up a secure pseudo distributed cluster
> - did some basic file system operations (mkdir, list, put, cat) and
> confirmed that everything was working
> - confirmed that the web UI worked
>
> Best,
> Aaron
>
> On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang 
> wrote:
>
> > Hi all,
> >
> > Let me start, as always, by thanking the efforts of all the contributors
> > who contributed to this release, especially those who jumped on the
> issues
> > found in RC0.
> >
> > I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
> > fixed JIRAs since the previous 3.0.0-beta1 release.
> >
> > You can find the artifacts here:
> >
> > http://home.apache.org/~wang/3.0.0-RC1/
> >
> > I've done the traditional testing of building from the source tarball and
> > running a Pi job on a single node cluster. I also verified that the
> shaded
> > jars are not empty.
> >
> > Found one issue that create-release (probably due to the mvn deploy
> change)
> > didn't sign the artifacts, but I fixed that by calling mvn one more time.
> > Available here:
> >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1075/
> >
> > This release will run the standard 5 days, closing on Dec 13th at 12:31pm
> > Pacific. My +1 to start.
> >
> > Best,
> > Andrew
> >
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Andrew Wang]

Sorry, forgot to push the tag. It's up there now.

On Sun, Dec 10, 2017 at 8:31 PM, Vinod Kumar Vavilapalli  wrote:

> I couldn't find the release tag for RC1 either - is it just me or has the
> release-process changed?
>
> +Vinod
>
> > On Dec 10, 2017, at 4:31 PM, Sangjin Lee  wrote:
> >
> > Hi Andrew,
> >
> > Thanks much for your effort! Just to be clear, could you please state the
> > git commit id of the RC1 we're voting for?
> >
> > Sangjin
> >
> > On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang 
> > wrote:
> >
> >> Hi all,
> >>
> >> Let me start, as always, by thanking the efforts of all the contributors
> >> who contributed to this release, especially those who jumped on the
> issues
> >> found in RC0.
> >>
> >> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
> >> fixed JIRAs since the previous 3.0.0-beta1 release.
> >>
> >> You can find the artifacts here:
> >>
> >> http://home.apache.org/~wang/3.0.0-RC1/
> >>
> >> I've done the traditional testing of building from the source tarball
> and
> >> running a Pi job on a single node cluster. I also verified that the
> shaded
> >> jars are not empty.
> >>
> >> Found one issue that create-release (probably due to the mvn deploy
> change)
> >> didn't sign the artifacts, but I fixed that by calling mvn one more
> time.
> >> Available here:
> >>
> >> https://repository.apache.org/content/repositories/
> orgapachehadoop-1075/
> >>
> >> This release will run the standard 5 days, closing on Dec 13th at
> 12:31pm
> >> Pacific. My +1 to start.
> >>
> >> Best,
> >> Andrew
> >>
>
>

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[Kihwal Lee]

+1 (binding)
- checked out the rc1 tag and built the source (-Pdist -Pnative)
- brought up a pseudo distributed cluster
- ran sample MR jobs
- verified web UIs working.

On Thu, Dec 7, 2017 at 9:22 PM, Konstantin Shvachko 
wrote:

> Hi everybody,
>
> I updated CHANGES.txt and fixed documentation links.
> Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
>
> This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
> previous one 2.7.4 was release August 4, 2017.
> Release 2.7.5 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
>
> Please give it a try and vote on this thread. The vote will run for 5 days
> ending 12/13/2017.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/

[Dec 11, 2017 2:31:46 AM] (wwei) YARN-7608. Incorrect sTarget column causing 
DataTable warning on RM
[Dec 11, 2017 1:50:02 PM] (sunilg) YARN-7632. Effective min and max resource 
need to be set for auto




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Possible null pointer dereference of replication in 
org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(BlockType,
 Short, Byte) Dereferenced at INodeFile.java:replication in 
org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(BlockType,
 Short, Byte) Dereferenced at INodeFile.java:[line 210] 

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 
   hadoop.hdfs.TestReplication 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.TestReadStripedFileWithDecoding 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.TestErasureCodingPolicies 
   hadoop.hdfs.TestDFSStripedOutputStream 
   hadoop.hdfs.server.namenode.TestDecommissioningStatus 
   hadoop.hdfs.TestReconstructStripedFile 
   
hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch 
   
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
 
   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched 
   hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart 
   hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator 
   hadoop.mapreduce.v2.TestUberAM 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/diff-compile-javac-root.txt
  [280K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/whitespace-eol.txt
  [8.8M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/whitespace-tabs.txt
  [288K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [380K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [44K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [100K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [20K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/619/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
  

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Aaron T. Myers]

+1 (binding)

- downloaded the src tarball and built the source (-Pdist -Pnative)
- verified the checksum
- brought up a secure pseudo distributed cluster
- did some basic file system operations (mkdir, list, put, cat) and
confirmed that everything was working
- confirmed that the web UI worked

Best,
Aaron

On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang 
wrote:

> Hi all,
>
> Let me start, as always, by thanking the efforts of all the contributors
> who contributed to this release, especially those who jumped on the issues
> found in RC0.
>
> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
> fixed JIRAs since the previous 3.0.0-beta1 release.
>
> You can find the artifacts here:
>
> http://home.apache.org/~wang/3.0.0-RC1/
>
> I've done the traditional testing of building from the source tarball and
> running a Pi job on a single node cluster. I also verified that the shaded
> jars are not empty.
>
> Found one issue that create-release (probably due to the mvn deploy change)
> didn't sign the artifacts, but I fixed that by calling mvn one more time.
> Available here:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1075/
>
> This release will run the standard 5 days, closing on Dec 13th at 12:31pm
> Pacific. My +1 to start.
>
> Best,
> Andrew
>

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Brahma Reddy Battula]

+1 (non-binding), thanks Junping for driving this.


--Built from the source
--Installaed 3 Node HA cluster
--Verified Basic shell Commands
--Browsed the HDFS/YARN web UI
--Ran sample pi,wordcount jobs

--Brahma Reddy Battula


On Tue, Dec 5, 2017 at 3:28 PM, Junping Du  wrote:

> Hi all,
>  I've created the first release candidate (RC0) for Apache Hadoop
> 2.8.3. This is our next maint release to follow up 2.8.2. It includes 79
> important fixes and improvements.
>
>   The RC artifacts are available at: http://home.apache.org/~
> junping_du/hadoop-2.8.3-RC0
>
>   The RC tag in git is: release-2.8.3-RC0
>
>   The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1072
>
>   Please try the release and vote; the vote will run for the usual 5
> working days, ending on 12/12/2017 PST time.
>
> Thanks,
>
> Junping
>



-- 



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Junping Du]

Hi Konstantin,

 Thanks for verification and comments. I was verifying your example below 
but found it is actually matched:


jduMBP:hadoop-2.8.3 jdu$ md5 ~/Downloads/hadoop-2.8.3-src.tar.gz
MD5 (/Users/jdu/Downloads/hadoop-2.8.3-src.tar.gz) = 
e53d04477b85e8b58ac0a26468f04736

What's your md5 checksum for given source tar ball?


Thanks,


Junping



From: Konstantin Shvachko 
Sent: Saturday, December 9, 2017 11:06 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

Hey Junping,

Could you pls upload mds relative to the tar.gz etc. files rather than their 
full path
/build/source/target/artifacts/hadoop-2.8.3-src.tar.gz:
   MD5 = E5 3D 04 47 7B 85 E8 B5  8A C0 A2 64 68 F0 47 36

Otherwise mds don't match for me.

Thanks,
--Konstantin

On Tue, Dec 5, 2017 at 1:58 AM, Junping Du 
> wrote:
Hi all,
 I've created the first release candidate (RC0) for Apache Hadoop 2.8.3. 
This is our next maint release to follow up 2.8.2. It includes 79 important 
fixes and improvements.

  The RC artifacts are available at: 
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0

  The RC tag in git is: release-2.8.3-RC0

  The maven artifacts are available via 
repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1072

  Please try the release and vote; the vote will run for the usual 5 
working days, ending on 12/12/2017 PST time.

Thanks,

Junping

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Junping Du]

Thanks Eric for verification!

A kindly reminder: The original due date for 2.8.3 voting is tomorrow but I 
only receive 1 binding vote so far - we have 76 PMCs and 127 Committers!

I can understand the whole community are busy with 3 release RC voting (2.7.5, 
2.8.3 and 3.0.0) and may be it is necessary to extend the voting period for a 
few more days. But please try as much as possible to verify our release bits. 
Thanks!


Thanks,


Junping?



From: Eric Payne 
Sent: Monday, December 11, 2017 1:51 PM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

Thanks Junping for the hard work on this release.

+1 (binding)

On a 6 node pseudo cluster (4 NMs), I performed the following manual tests:

- Built and installed from source

- Successfully ran a stream job

- Verified that user weights are honored by assigning the appropriate amount of 
resources to the weighted users.

- Ensured that FiarOrderingPolicy and FifoOrderingPolicy worked in the Capacity 
Scheduler as expected

- Applications with higher priorities are assigned containers as expected in 
the FifoOrderingPolicy of the Capacity Scheduler until the user reaches its 
user resource limit.

Eric Payne



From: Junping Du 
To: "common-...@hadoop.apache.org" ; 
"hdfs-...@hadoop.apache.org" ; 
"mapreduce-...@hadoop.apache.org" ; 
"yarn-dev@hadoop.apache.org" 
Sent: Tuesday, December 5, 2017 3:58 AM
Subject: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

Hi all,
I've created the first release candidate (RC0) for Apache Hadoop 2.8.3. 
This is our next maint release to follow up 2.8.2. It includes 79 important 
fixes and improvements.

  The RC artifacts are available at: 
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0

  The RC tag in git is: release-2.8.3-RC0

  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1072

  Please try the release and vote; the vote will run for the usual 5 
working days, ending on 12/12/2017 PST time.

Thanks,

Junping

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Andrew Wang]

Good point on the mutability. Release tags are immutable, RCs are not.

On Mon, Dec 11, 2017 at 1:39 PM, Sangjin Lee  wrote:

> Thanks Andrew. For the record, the commit id would be
> c25427ceca461ee979d30edd7a4b0f50718e6533. I mention that for completeness
> because of the mutability of tags.
>
> On Mon, Dec 11, 2017 at 10:31 AM, Andrew Wang 
> wrote:
>
>> Sorry, forgot to push the tag. It's up there now.
>>
>> On Sun, Dec 10, 2017 at 8:31 PM, Vinod Kumar Vavilapalli <
>> vino...@apache.org> wrote:
>>
>>> I couldn't find the release tag for RC1 either - is it just me or has
>>> the release-process changed?
>>>
>>> +Vinod
>>>
>>> > On Dec 10, 2017, at 4:31 PM, Sangjin Lee  wrote:
>>> >
>>> > Hi Andrew,
>>> >
>>> > Thanks much for your effort! Just to be clear, could you please state
>>> the
>>> > git commit id of the RC1 we're voting for?
>>> >
>>> > Sangjin
>>> >
>>> > On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang >> >
>>> > wrote:
>>> >
>>> >> Hi all,
>>> >>
>>> >> Let me start, as always, by thanking the efforts of all the
>>> contributors
>>> >> who contributed to this release, especially those who jumped on the
>>> issues
>>> >> found in RC0.
>>> >>
>>> >> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates
>>> 302
>>> >> fixed JIRAs since the previous 3.0.0-beta1 release.
>>> >>
>>> >> You can find the artifacts here:
>>> >>
>>> >> http://home.apache.org/~wang/3.0.0-RC1/
>>> >>
>>> >> I've done the traditional testing of building from the source tarball
>>> and
>>> >> running a Pi job on a single node cluster. I also verified that the
>>> shaded
>>> >> jars are not empty.
>>> >>
>>> >> Found one issue that create-release (probably due to the mvn deploy
>>> change)
>>> >> didn't sign the artifacts, but I fixed that by calling mvn one more
>>> time.
>>> >> Available here:
>>> >>
>>> >> https://repository.apache.org/content/repositories/orgapache
>>> hadoop-1075/
>>> >>
>>> >> This release will run the standard 5 days, closing on Dec 13th at
>>> 12:31pm
>>> >> Pacific. My +1 to start.
>>> >>
>>> >> Best,
>>> >> Andrew
>>> >>
>>>
>>>
>>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Sangjin Lee]

Thanks Andrew. For the record, the commit id would be
c25427ceca461ee979d30edd7a4b0f50718e6533. I mention that for completeness
because of the mutability of tags.

On Mon, Dec 11, 2017 at 10:31 AM, Andrew Wang 
wrote:

> Sorry, forgot to push the tag. It's up there now.
>
> On Sun, Dec 10, 2017 at 8:31 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> I couldn't find the release tag for RC1 either - is it just me or has the
>> release-process changed?
>>
>> +Vinod
>>
>> > On Dec 10, 2017, at 4:31 PM, Sangjin Lee  wrote:
>> >
>> > Hi Andrew,
>> >
>> > Thanks much for your effort! Just to be clear, could you please state
>> the
>> > git commit id of the RC1 we're voting for?
>> >
>> > Sangjin
>> >
>> > On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang 
>> > wrote:
>> >
>> >> Hi all,
>> >>
>> >> Let me start, as always, by thanking the efforts of all the
>> contributors
>> >> who contributed to this release, especially those who jumped on the
>> issues
>> >> found in RC0.
>> >>
>> >> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates
>> 302
>> >> fixed JIRAs since the previous 3.0.0-beta1 release.
>> >>
>> >> You can find the artifacts here:
>> >>
>> >> http://home.apache.org/~wang/3.0.0-RC1/
>> >>
>> >> I've done the traditional testing of building from the source tarball
>> and
>> >> running a Pi job on a single node cluster. I also verified that the
>> shaded
>> >> jars are not empty.
>> >>
>> >> Found one issue that create-release (probably due to the mvn deploy
>> change)
>> >> didn't sign the artifacts, but I fixed that by calling mvn one more
>> time.
>> >> Available here:
>> >>
>> >> https://repository.apache.org/content/repositories/orgapache
>> hadoop-1075/
>> >>
>> >> This release will run the standard 5 days, closing on Dec 13th at
>> 12:31pm
>> >> Pacific. My +1 to start.
>> >>
>> >> Best,
>> >> Andrew
>> >>
>>
>>
>

[jira] [Created] (YARN-7639) Queue Management scheduling edit policy class needs to be configured dynamically

[Suma Shivaprasad (JIRA)]

Suma Shivaprasad created YARN-7639:
--

 Summary: Queue Management scheduling edit policy class needs to be 
configured dynamically
 Key: YARN-7639
 URL: https://issues.apache.org/jira/browse/YARN-7639
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Suma Shivaprasad
Assignee: Suma Shivaprasad


This needs to be configured dynamically for 
yarn.resourcemanager.monitor.capacity.queue-management.monitoring-interval 
whenever auto leaf queue creation is enabled for a parent queue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7638) Add unit tests for Preemption and Recovery

[Suma Shivaprasad (JIRA)]

Suma Shivaprasad created YARN-7638:
--

 Summary: Add unit tests for Preemption and Recovery
 Key: YARN-7638
 URL: https://issues.apache.org/jira/browse/YARN-7638
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Suma Shivaprasad
Assignee: Suma Shivaprasad


Add unit tests to test inter leaf-queue pre-emption based on utilization and 
work preserving start/recovery.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7637) GPU volume creation command fails when work preserving is disabled at NM

[Sunil G (JIRA)]

Sunil G created YARN-7637:
-

 Summary: GPU volume creation command fails when work preserving is 
disabled at NM
 Key: YARN-7637
 URL: https://issues.apache.org/jira/browse/YARN-7637
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: nodemanager
Affects Versions: 3.1.0
Reporter: Sunil G
Assignee: Zian Chen


When work preserving is disabled, NM uses {{NMNullStateStoreService}}. Hence 
resource mappings related to GPU wont be saved at Container.

This has to  be rechecked and store accordingly.

cc/ [~leftnoteasy] and [~Zian Chen]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-7534) Fair scheduler assign resources may exceed maxResources

[Wilfred Spiegelenburg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wilfred Spiegelenburg resolved YARN-7534.
-
Resolution: Cannot Reproduce

No issue found the code shows that we check the queue size in the FS and we 
have no logs that show this is not working

> Fair scheduler assign resources may exceed maxResources
> ---
>
> Key: YARN-7534
> URL: https://issues.apache.org/jira/browse/YARN-7534
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Reporter: YunFan Zhou
>Assignee: Wilfred Spiegelenburg
>
> The logic we're scheduling now is to check whether the resources used by the 
> queue has exceeded *maxResources* before assigning the container. This will 
> leads to the fact that after assigning this container the queue uses more 
> resources than *maxResources*.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7636) Re-reservation count may overflow when cluster resource exhausted for a long time

[Tao Yang (JIRA)]

Tao Yang created YARN-7636:
--

 Summary: Re-reservation count may overflow when cluster resource 
exhausted for a long time 
 Key: YARN-7636
 URL: https://issues.apache.org/jira/browse/YARN-7636
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacityscheduler
Affects Versions: 3.0.0-alpha4, 2.9.1
Reporter: Tao Yang
Assignee: Tao Yang


Exception stack:
{noformat}
java.lang.IllegalArgumentException: Overflow adding 1 occurrences to a count of 
2147483647
        at 
com.google.common.collect.ConcurrentHashMultiset.add(ConcurrentHashMultiset.java:246)
        at 
com.google.common.collect.AbstractMultiset.add(AbstractMultiset.java:80)
        at 
com.google.common.collect.ConcurrentHashMultiset.add(ConcurrentHashMultiset.java:51)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.addReReservation(SchedulerApplicationAttempt.java:406)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.reserve(SchedulerApplicationAttempt.java:555)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.reserve(FiCaSchedulerApp.java:1076)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.apply(FiCaSchedulerApp.java:795)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.tryCommit(CapacityScheduler.java:2770)
        at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$ResourceCommitterService.run(CapacityScheduler.java:546)
{noformat}
We can add check condition {{getReReservations(schedulerKey) < 
Integer.MAX_VALUE}} before  addReReservation to avoid this problem.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org