[jira] [Comment Edited] (YARN-8992) Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue

2018-11-08 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680630#comment-16680630 ] Haibo Chen edited comment on YARN-8992 at 11/9/18 12:01 AM: Se

[jira] [Updated] (YARN-8992) Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue

2018-11-08 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8992: - Description: As discovered in YARN-8990, QueueManager can see a leaf queue being empty while FSLeafQueue.

[jira] [Commented] (YARN-8990) FS: race condition in app submit and queue cleanup

2018-11-08 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680628#comment-16680628 ] Haibo Chen commented on YARN-8990: -- Okay. I'll just do that. +1 on 001 patch. Will check

[jira] [Commented] (YARN-8990) FS: race condition in app submit and queue cleanup

2018-11-08 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680620#comment-16680620 ] Haibo Chen commented on YARN-8990: -- Another option is to split out the new race condition

[jira] [Commented] (YARN-8990) FS: race condition in app submit and queue cleanup

2018-11-08 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680481#comment-16680481 ] Haibo Chen commented on YARN-8990: -- Thanks [~wilfreds] for the patch!  I have taken the l

[jira] [Updated] (YARN-8990) FS: race condition in app submit and queue cleanup

2018-11-08 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8990: - Attachment: YARN-8990.002.patch > FS: race condition in app submit and queue cleanup > ---

[jira] [Commented] (YARN-8932) ResourceUtilization cpu is misused in oversubscription as a percentage

2018-11-02 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673638#comment-16673638 ] Haibo Chen commented on YARN-8932: -- The shaded client issue is unrelated to this patch. I

[jira] [Updated] (YARN-8932) ResourceUtilization cpu is misused in oversubscription as a percentage

2018-10-30 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8932: - Attachment: YARN-8932-YARN-1011.02.patch > ResourceUtilization cpu is misused in oversubscription as a per

[jira] [Commented] (YARN-8932) ResourceUtilization cpu is misused in oversubscription as a percentage

2018-10-30 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669331#comment-16669331 ] Haibo Chen commented on YARN-8932: -- Thanks for the review, [~rkanter]. I have updated the

[jira] [Commented] (YARN-8932) ResourceUtilization cpu is misused in oversubscription as a percentage

2018-10-30 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669151#comment-16669151 ] Haibo Chen commented on YARN-8932: -- The unit test failure is unrelated. The code changes

[jira] [Commented] (YARN-8932) ResourceUtilization cpu is misused in oversubscription as a percentage

2018-10-27 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1181#comment-1181 ] Haibo Chen commented on YARN-8932: -- Patch updated to address the checkstyle issues and th

[jira] [Updated] (YARN-8932) ResourceUtilization cpu is misused in oversubscription as a percentage

2018-10-27 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8932: - Attachment: YARN-8932-YARN-1011.01.patch > ResourceUtilization cpu is misused in oversubscription as a per

[jira] [Updated] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers

2018-10-27 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-1011: - Attachment: (was: YARN-8932-YARN-1011.01.patch) > [Umbrella] Schedule containers based on utilization

[jira] [Updated] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers

2018-10-27 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-1011: - Attachment: YARN-8932-YARN-1011.01.patch > [Umbrella] Schedule containers based on utilization of currentl

[jira] [Updated] (YARN-8932) ResourceUtilization cpu is misused in oversubscription as a percentage

2018-10-26 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8932: - Attachment: YARN-8932-YARN-1011.00.patch > ResourceUtilization cpu is misused in oversubscription as a per

[jira] [Updated] (YARN-8921) SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs

2018-10-25 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8921: - Attachment: YARN-8921-YARN-1011.02.patch > SnapshotBasedOverAllocationPolicy always caps the amount of mem

[jira] [Commented] (YARN-8921) SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs

2018-10-25 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16664217#comment-16664217 ] Haibo Chen commented on YARN-8921: -- Oops, I forgot to include the unit test file into the

[jira] [Assigned] (YARN-8470) Fair scheduler exception with SLS

2018-10-25 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen reassigned YARN-8470: Assignee: (was: Haibo Chen) > Fair scheduler exception with SLS > -

[jira] [Assigned] (YARN-6356) Allow different values of yarn.log-aggregation.retain-seconds for succeeded and failed jobs

2018-10-25 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen reassigned YARN-6356: Assignee: (was: Haibo Chen) > Allow different values of yarn.log-aggregation.retain-seconds for

[jira] [Commented] (YARN-8921) SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs

2018-10-25 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663961#comment-16663961 ] Haibo Chen commented on YARN-8921: -- Thanks [~rkanter] for the review.  I have updated the

[jira] [Updated] (YARN-8921) SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs

2018-10-25 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8921: - Attachment: YARN-8921-YARN-1011.01.patch > SnapshotBasedOverAllocationPolicy always caps the amount of mem

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-24 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662393#comment-16662393 ] Haibo Chen commented on YARN-8911: -- Thanks [~elgoiri] for the review! I have checked 02 p

[jira] [Commented] (YARN-8921) SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs

2018-10-23 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661426#comment-16661426 ] Haibo Chen commented on YARN-8921: -- Unit test failure is unrelated. > SnapshotBasedOverA

[jira] [Commented] (YARN-8929) DefaultOOMHandler should only pick running containers to kill upon oom events

2018-10-23 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661414#comment-16661414 ] Haibo Chen commented on YARN-8929: -- Thanks [~rkanter] for the review. The unit test failu

[jira] [Updated] (YARN-8929) DefaultOOMHandler should only pick running containers to kill upon oom events

2018-10-23 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8929: - Attachment: YARN-8929.01.patch > DefaultOOMHandler should only pick running containers to kill upon oom ev

[jira] [Updated] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-23 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8911: - Attachment: YARN-8911.02.patch > ContainerScheduler incorrectly uses percentage number as the cpu resource

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-23 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661341#comment-16661341 ] Haibo Chen commented on YARN-8911: -- Good point. I have modified TestContainerSchedulerRec

[jira] [Commented] (YARN-8059) Resource type is ignored when FS decide to preempt

2018-10-23 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661082#comment-16661082 ] Haibo Chen commented on YARN-8059: -- Not a big fan of have a boolean check in all most eve

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-23 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660988#comment-16660988 ] Haibo Chen commented on YARN-8911: -- {quote} Is there any test that would track this from

[jira] [Updated] (YARN-8930) CGroup-based strict container memory enforcement does not work with CGroupElasticMemoryController

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8930: - Attachment: YARN-8930.01.patch > CGroup-based strict container memory enforcement does not work with > CG

[jira] [Commented] (YARN-8930) CGroup-based strict container memory enforcement does not work with CGroupElasticMemoryController

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659812#comment-16659812 ] Haibo Chen commented on YARN-8930: -- Cleaned up the checkstyle issues in the new patch. >

[jira] [Commented] (YARN-8930) CGroup-based strict container memory enforcement does not work with CGroupElasticMemoryController

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659761#comment-16659761 ] Haibo Chen commented on YARN-8930: -- The patch allows the polling-based memory check to ki

[jira] [Updated] (YARN-8932) ResourceUtilization cpu is misused in oversubscription as a percentage

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8932: - Summary: ResourceUtilization cpu is misused in oversubscription as a percentage (was: ResourceUtilization

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659739#comment-16659739 ] Haibo Chen commented on YARN-8911: -- Uploaded a new patch that 1) updates the ResourceUti

[jira] [Updated] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8911: - Attachment: YARN-8911.01.patch > ContainerScheduler incorrectly uses percentage number as the cpu resource

[jira] [Updated] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8911: - Description: *UPDATE*:  *per discussion below, the cpu resource utlization (ResourceUtilzation.cpu) is in

[jira] [Updated] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8911: - Summary: ContainerScheduler incorrectly uses percentage number as the cpu resource utlization (was: NM in

[jira] [Created] (YARN-8932) ResourceUtilization cpu is misused in oversubscription

2018-10-22 Thread Haibo Chen (JIRA)
Haibo Chen created YARN-8932: Summary: ResourceUtilization cpu is misused in oversubscription Key: YARN-8932 URL: https://issues.apache.org/jira/browse/YARN-8932 Project: Hadoop YARN Issue Type:

[jira] [Updated] (YARN-8930) CGroup-based strict container memory enforcement does not work with CGroupElasticMemoryController

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8930: - Attachment: YARN-8930.00.patch > CGroup-based strict container memory enforcement does not work with > CG

[jira] [Commented] (YARN-8929) DefaultOOMHandler should only pick running containers to kill upon oom events

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659526#comment-16659526 ] Haibo Chen commented on YARN-8929: -- The patch does three things 1) To kill a container,

[jira] [Updated] (YARN-8930) CGroup-based strict container memory enforcement does not work with CGroupElasticMemoryController

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8930: - Summary: CGroup-based strict container memory enforcement does not work with CGroupElasticMemoryController

[jira] [Created] (YARN-8930) CGroup-based strict container memory management does not work with CGroupElasticMemoryController

2018-10-22 Thread Haibo Chen (JIRA)
Haibo Chen created YARN-8930: Summary: CGroup-based strict container memory management does not work with CGroupElasticMemoryController Key: YARN-8930 URL: https://issues.apache.org/jira/browse/YARN-8930

[jira] [Updated] (YARN-8929) DefaultOOMHandler should only pick running containers to kill upon oom events

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8929: - Attachment: YARN-8929.00.patch > DefaultOOMHandler should only pick running containers to kill upon oom ev

[jira] [Updated] (YARN-8929) DefaultOOMHandler should only pick running containers to kill upon oom events

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8929: - Attachment: (was: YARN-8921-YARN-1011.00.patch) > DefaultOOMHandler should only pick running container

[jira] [Updated] (YARN-8929) DefaultOOMHandler should only pick running containers to kill upon oom events

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8929: - Attachment: YARN-8921-YARN-1011.00.patch > DefaultOOMHandler should only pick running containers to kill u

[jira] [Updated] (YARN-8921) SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8921: - Attachment: YARN-8921-YARN-1011.00.patch > SnapshotBasedOverAllocationPolicy always caps the amount of mem

[jira] [Commented] (YARN-8921) SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659263#comment-16659263 ] Haibo Chen commented on YARN-8921: -- oops, forgot to include the branch name in the patch

[jira] [Created] (YARN-8929) DefaultOOMHandler should only pick running containers to kill upon oom events

2018-10-22 Thread Haibo Chen (JIRA)
Haibo Chen created YARN-8929: Summary: DefaultOOMHandler should only pick running containers to kill upon oom events Key: YARN-8929 URL: https://issues.apache.org/jira/browse/YARN-8929 Project: Hadoop YAR

[jira] [Updated] (YARN-8921) SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8921: - Attachment: YARN-8921.00.patch > SnapshotBasedOverAllocationPolicy always caps the amount of memory availa

[jira] [Updated] (YARN-8921) SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs

2018-10-22 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8921: - Summary: SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs (was: Snaps

[jira] [Commented] (YARN-8911) NM incorrectly account for container cpu utilization by their number of vcores

2018-10-19 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657594#comment-16657594 ] Haibo Chen commented on YARN-8911: -- I see.  I can update YARN-1011 oversubscription code

[jira] [Commented] (YARN-8911) NM incorrectly account for container cpu utilization by their number of vcores

2018-10-19 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657488#comment-16657488 ] Haibo Chen commented on YARN-8911: -- CC [~elgoiri], who wrote the initial code. > NM inco

[jira] [Commented] (YARN-8911) NM incorrectly account for container cpu utilization by their number of vcores

2018-10-19 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657482#comment-16657482 ] Haibo Chen commented on YARN-8911: -- YARN-1011 has been using the cpu portion of ResourceU

[jira] [Comment Edited] (YARN-8911) NM incorrectly account for container cpu utilization by their number of vcores

2018-10-19 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657482#comment-16657482 ] Haibo Chen edited comment on YARN-8911 at 10/19/18 9:48 PM: YA

[jira] [Updated] (YARN-8921) SnapshotBasedOverAllocationPolicy incorrectly caps the amount of memory availabe in bytes to Integer.MAX_VALUE

2018-10-19 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8921: - Summary: SnapshotBasedOverAllocationPolicy incorrectly caps the amount of memory availabe in bytes to Inte

[jira] [Created] (YARN-8921) SnapshotBasedOverAllocationPolicy incorrectly rounds memory availabe int bytes

2018-10-19 Thread Haibo Chen (JIRA)
Haibo Chen created YARN-8921: Summary: SnapshotBasedOverAllocationPolicy incorrectly rounds memory availabe int bytes Key: YARN-8921 URL: https://issues.apache.org/jira/browse/YARN-8921 Project: Hadoop YA

[jira] [Updated] (YARN-8449) RM HA for AM web server HTTPS Support

2018-10-18 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8449: - Summary: RM HA for AM web server HTTPS Support (was: RM HA for AM HTTPS Support) > RM HA for AM web serv

[jira] [Commented] (YARN-8911) NM incorrectly account for container cpu utilization by their number of vcores

2018-10-18 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655859#comment-16655859 ] Haibo Chen commented on YARN-8911: -- CC [~asuresh]  We found this during functional testin

[jira] [Updated] (YARN-8911) NM incorrectly account for container cpu utilization by their number of vcores

2018-10-18 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8911: - Attachment: YARN-8911.00.patch > NM incorrectly account for container cpu utilization by their number of v

[jira] [Updated] (YARN-8911) NM incorrectly account for container cpu utilization by their number of vcores

2018-10-18 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8911: - Description: ResourceUtilization represents the cpu utilization with a float number in [0, 1.0], i.e. the

[jira] [Created] (YARN-8911) NM incorrectly account for container cpu utilization by their number of vcores

2018-10-18 Thread Haibo Chen (JIRA)
Haibo Chen created YARN-8911: Summary: NM incorrectly account for container cpu utilization by their number of vcores Key: YARN-8911 URL: https://issues.apache.org/jira/browse/YARN-8911 Project: Hadoop YA

[jira] [Commented] (YARN-8864) NM incorrectly logs container user as the user who sent a start/stop container request in its audit log

2018-10-18 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655446#comment-16655446 ] Haibo Chen commented on YARN-8864: -- +1. The native test failures was seen YARN-8448. Chec

[jira] [Commented] (YARN-8449) RM HA for AM HTTPS Support

2018-10-18 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655436#comment-16655436 ] Haibo Chen commented on YARN-8449: -- [~rkanter] can you address the outstanding checkstyle

[jira] [Commented] (YARN-8864) NM incorrectly logs container user as the user who sent a start/stop container request in its audit log

2018-10-17 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654466#comment-16654466 ] Haibo Chen commented on YARN-8864: -- +1 pending jenkins. > NM incorrectly logs container

[jira] [Commented] (YARN-8449) RM HA for AM HTTPS Support

2018-10-17 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654144#comment-16654144 ] Haibo Chen commented on YARN-8449: -- +1 on the latest (002) patch pending Jenkins. > RM H

[jira] [Updated] (YARN-8582) Document YARN support for HTTPS in AM Web server

2018-10-16 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8582: - Summary: Document YARN support for HTTPS in AM Web server (was: Documentation for AM HTTPS Support) > Do

[jira] [Commented] (YARN-8582) Documentation for AM HTTPS Support

2018-10-16 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652714#comment-16652714 ] Haibo Chen commented on YARN-8582: -- +1 on the latest patch. Checking it in shortly. > Do

[jira] [Commented] (YARN-8449) RM HA for AM HTTPS Support

2018-10-16 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652710#comment-16652710 ] Haibo Chen commented on YARN-8449: -- Thanks [~rkanter] for the patch. I have just one mino

[jira] [Updated] (YARN-8842) Expose metrics for custom resource types in QueueMetrics

2018-10-16 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8842: - Summary: Expose metrics for custom resource types in QueueMetrics (was: Update QueueMetrics with custom r

[jira] [Commented] (YARN-8842) Update QueueMetrics with custom resource values

2018-10-16 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652443#comment-16652443 ] Haibo Chen commented on YARN-8842: -- +1 on the latest patch. I'll fix the one minor indent

[jira] [Updated] (YARN-8448) AM HTTPS Support for AM communication with RMWeb proxy

2018-10-16 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8448: - Summary: AM HTTPS Support for AM communication with RMWeb proxy (was: AM HTTPS Support) > AM HTTPS Suppo

[jira] [Commented] (YARN-8448) AM HTTPS Support

2018-10-16 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652392#comment-16652392 ] Haibo Chen commented on YARN-8448: -- I ran the cestest locally and it did not fail for me

[jira] [Commented] (YARN-8448) AM HTTPS Support

2018-10-15 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650587#comment-16650587 ] Haibo Chen commented on YARN-8448: -- Thanks [~rkanter] for addressing the comments!  {quot

[jira] [Updated] (YARN-8775) TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File modifications

2018-10-15 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8775: - Labels: unittest (was: ) > TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File

[jira] [Updated] (YARN-8775) TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File modifications

2018-10-15 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8775: - Description: The test can fail sometimes when file operations were done during the disk health check done

[jira] [Commented] (YARN-8775) TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File modifications

2018-10-15 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650457#comment-16650457 ] Haibo Chen commented on YARN-8775: -- +1 on the latest patch. Will check it in shortly. Tha

[jira] [Commented] (YARN-8842) Update QueueMetrics with custom resource values

2018-10-15 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650450#comment-16650450 ] Haibo Chen commented on YARN-8842: -- Thanks [~snemeth] for updating the patch!  The patch

[jira] [Commented] (YARN-8448) AM HTTPS Support

2018-10-12 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648434#comment-16648434 ] Haibo Chen commented on YARN-8448: -- For the ProxyCA related changes, I have a few questio

[jira] [Commented] (YARN-8448) AM HTTPS Support

2018-10-12 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648293#comment-16648293 ] Haibo Chen commented on YARN-8448: -- A few minor comments/questions about the c code chang

[jira] [Created] (YARN-8874) NM does not do any authorization in ContainerManagerImpl.signalToContainer()

2018-10-12 Thread Haibo Chen (JIRA)
Haibo Chen created YARN-8874: Summary: NM does not do any authorization in ContainerManagerImpl.signalToContainer() Key: YARN-8874 URL: https://issues.apache.org/jira/browse/YARN-8874 Project: Hadoop YARN

[jira] [Commented] (YARN-8864) NM incorrectly logs container user as the user who sent a stop container request in its audit log

2018-10-12 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648077#comment-16648077 ] Haibo Chen commented on YARN-8864: -- Thanks [~wilfreds] for the patch!  I believe the user

[jira] [Updated] (YARN-8864) NM incorrectly logs container user as the user who sent a start/stop container request in its audit log

2018-10-12 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8864: - Summary: NM incorrectly logs container user as the user who sent a start/stop container request in its aud

[jira] [Comment Edited] (YARN-8775) TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File modifications

2018-10-12 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648045#comment-16648045 ] Haibo Chen edited comment on YARN-8775 at 10/12/18 3:44 PM: Th

[jira] [Commented] (YARN-8775) TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File modifications

2018-10-12 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648045#comment-16648045 ] Haibo Chen commented on YARN-8775: -- Thanks for the update, [~bsteinbach]. A few comments

[jira] [Commented] (YARN-8448) AM HTTPS Support

2018-10-11 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647209#comment-16647209 ] Haibo Chen commented on YARN-8448: -- Thanks [~rkanter] for the patch update! Posting some

[jira] [Commented] (YARN-8448) AM HTTPS Support

2018-10-11 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647210#comment-16647210 ] Haibo Chen commented on YARN-8448: -- I'll continue the review tomorrow and post remaining

[jira] [Commented] (YARN-8775) TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File modifications

2018-10-11 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646855#comment-16646855 ] Haibo Chen commented on YARN-8775: -- I see.  That is indeed an issue. Thinking about this

[jira] [Updated] (YARN-8864) NM incorrectly logs container user as the user who sent a stop container request in its audit log

2018-10-09 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8864: - Description: As in  ContainerManagerImpl.java {code:java} protected void stopContainerInternal(ContainerId

[jira] [Created] (YARN-8864) NM incorrectly logs container user as the user who sent a stop container request in its audit log

2018-10-09 Thread Haibo Chen (JIRA)
Haibo Chen created YARN-8864: Summary: NM incorrectly logs container user as the user who sent a stop container request in its audit log Key: YARN-8864 URL: https://issues.apache.org/jira/browse/YARN-8864

[jira] [Commented] (YARN-8807) FairScheduler crashes RM with oversubscription turned on if an application is killed.

2018-10-09 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644132#comment-16644132 ] Haibo Chen commented on YARN-8807: -- Thanks [~rkanter], and [~zsiegl] for the review! > F

[jira] [Commented] (YARN-8813) Improve debug messages for NM preemption of OPPORTUNISTIC containers

2018-10-09 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644096#comment-16644096 ] Haibo Chen commented on YARN-8813: -- That makes sense to me now. I have updated the patch

[jira] [Updated] (YARN-8813) Improve debug messages for NM preemption of OPPORTUNISTIC containers

2018-10-09 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8813: - Attachment: YARN-8813-YARN-1011.02.patch > Improve debug messages for NM preemption of OPPORTUNISTIC c

[jira] [Commented] (YARN-8448) AM HTTPS Support

2018-10-09 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643719#comment-16643719 ] Haibo Chen commented on YARN-8448: -- Thanks [~rkanter] for the elaboration. OFF is what YA

[jira] [Commented] (YARN-8807) FairScheduler crashes RM with oversubscription turned on if an application is killed.

2018-10-09 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643638#comment-16643638 ] Haibo Chen commented on YARN-8807: -- The unit test failures are all unrelated. > FairSche

[jira] [Commented] (YARN-8813) Improve debug messages for NM preemption of OPPORTUNISTIC containers

2018-10-09 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643633#comment-16643633 ] Haibo Chen commented on YARN-8813: -- The unit test failure is unrelated. The checkstyle do

[jira] [Updated] (YARN-8813) Improve debug messages for NM preemption of OPPORTUNISTIC containers

2018-10-08 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8813: - Attachment: YARN-8813-YARN-1011.01.patch > Improve debug messages for NM preemption of OPPORTUNISTIC c

[jira] [Commented] (YARN-8813) Improve debug messages for NM preemption of OPPORTUNISTIC containers

2018-10-08 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642623#comment-16642623 ] Haibo Chen commented on YARN-8813: -- Thanks [~rkanter] for your review. I addressed your c

[jira] [Commented] (YARN-8807) FairScheduler crashes RM with oversubscription turned on if an application is killed.

2018-10-08 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642621#comment-16642621 ] Haibo Chen commented on YARN-8807: -- Thanks for the review, [~rkanter]. I updated the patc

[jira] [Updated] (YARN-8807) FairScheduler crashes RM with oversubscription turned on if an application is killed.

2018-10-08 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8807: - Attachment: YARN-8807-YARN-1011.01.patch > FairScheduler crashes RM with oversubscription turned on if an

[jira] [Commented] (YARN-8448) AM HTTPS Support

2018-10-08 Thread Haibo Chen (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642603#comment-16642603 ] Haibo Chen commented on YARN-8448: -- Thanks [~rkanter]  for the the patch. I took a high-l

<    1   2   3   4   5   6   7   8   9   10   >