date:20170916

[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts

2017-09-16 Thread Eric Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169188#comment-16169188
 ] 

Eric Yang commented on YARN-5534:
-

[~miklos.szeg...@cloudera.com] It's a cute perspective, but there might be 
usability issues.  Today, it is possible to keep container-executor.cfg read 
only to root and yarn user.  Authorized and banned users are only known to root 
user and yarn user.  This is similar to sudoers file that managers who has 
sudoers rights.  

Other the other hand, file system mount points needs to be known by all users 
who would like to use mount points.  It would be more convenient to give 
everyone read access to file system mount point file, like /etc/fstab.  

If volume white list is mixed with user privileges control, then we lose some 
flexibility to keep banned users a secret or we lose ability to know what mount 
points can be used.  With this reason, I prefer to keep white list volume 
separated from container-executor.cfg for separation of duty from security 
point of view.
However, black list volume maintained in container-executor.cfg, can make 
attack more difficult.

> Allow whitelisted volume mounts 
> 
>
> Key: YARN-5534
> URL: https://issues.apache.org/jira/browse/YARN-5534
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: luhuichun
>Assignee: Shane Kumpf
> Attachments: YARN-5534.001.patch, YARN-5534.002.patch, 
> YARN-5534.003.patch
>
>
> Introduction 
> Mounting files or directories from the host is one way of passing 
> configuration and other information into a docker container. 
> We could allow the user to set a list of mounts in the environment of 
> ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). 
> These would be mounted read-only to the specified target locations. This has 
> been resolved in YARN-4595
> 2.Problem Definition
> Bug mounting arbitrary volumes into a Docker container can be a security risk.
> 3.Possible solutions
> one approach to provide safe mounts is to allow the cluster administrator to 
> configure a set of parent directories as white list mounting directories.
>  Add a property named yarn.nodemanager.volume-mounts.white-list, when 
> container executor do mount checking, only the allowed directories or 
> sub-directories can be mounted. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-6475) Fix some long function checkstyle issues

2017-09-16 Thread Soumabrata Chakraborty (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169017#comment-16169017
 ] 

Soumabrata Chakraborty edited comment on YARN-6475 at 9/17/17 3:54 AM:
---

[~asuresh]  [~miklos.szeg...@cloudera.com]  [~templedf]

Pull request https://github.com/apache/hadoop/pull/275 opened.
Patch for branch-2 has been attached to this JIRA.  


was (Author: soumabrata):
[~asuresh]  [~miklos.szeg...@cloudera.com]  [~templedf]

Patch for branch-2 has been attached to this JIRA.

> Fix some long function checkstyle issues
> 
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6475.001.patch, YARN-6475-branch-2.001.patch
>
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
>   @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
>   @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7193) Implement REST API for register Yarnfile in application catalog

2017-09-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169147#comment-16169147
 ] 

Hadoop QA commented on YARN-7193:
-

(!) A patch to the testing environment has been detected. 
Re-executing against the patched versions to perform further tests. 
The console is at 
https://builds.apache.org/job/PreCommit-YARN-Build/17494/console in case of 
problems.


> Implement REST API for register Yarnfile in application catalog
> ---
>
> Key: YARN-7193
> URL: https://issues.apache.org/jira/browse/YARN-7193
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: applications
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7193.yarn-native-services.001.patch, 
> YARN-7193.yarn-native-services.002.patch, 
> YARN-7193.yarn-native-services.003.patch, 
> YARN-7193.yarn-native-services.004.patch, 
> YARN-7193.yarn-native-services.005.patch
>
>
> For support ability to register and index Yarnfile, we need a set of REST API 
> to register, search, and recommend applications from application catalog.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor

2017-09-16 Thread Shane Kumpf (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169141#comment-16169141
 ] 

Shane Kumpf commented on YARN-6623:
---

Thanks for the patch [~vvasudev]! Overall it looks great. Thanks for the 
effort. Couple of comments.

Can we print out the mount that was requested but failed for the read-only and 
read write checks? I found it a bit difficult to determine which mount was the 
cause of the failure. Doing the same for the devices option would be helpful as 
well. Something like the following:
{code}
if (permitted_rw == 0) {
  fprintf(ERRORFILE, "Requested rw mount not found in the rw whitelist 
'%s', realpath=%s\n", values[i], mount_src);
  ret = INVALID_DOCKER_RW_MOUNT;
  goto free_and_exit;
-snip-
 if (ro != 0 && permitted_ro == 0 && permitted_rw == 0) {
fprintf(ERRORFILE, "Requested ro mount not found in the ro whitelist 
'%s', realpath=%s\n", values[i], mount_src);
ret = INVALID_DOCKER_RO_MOUNT;
goto free_and_exit;
  }
{code}

Currently only {{docker run}} works with this patch. I know you are aware of 
this. There are a few regressions in this patch from what was fixed in 
YARN-6726. It would be great if we could try to address them here. I'll list 
the ones I'm aware of below.

1) get_docker_inspect_command uses validate_container_id directly, instead of 
validate_container_name.
{code}
  if (container_name == NULL || validate_container_id(container_name) != 0) {
return INVALID_DOCKER_CONTAINER_NAME;
  }
{code}

2) The {{--format}} option for {{docker inspect}} can't be quoted, or it breaks 
the returned output.
{code}
quote_and_append_arg(_buffer, _buffer_size, " --format=", format);
{code}

3) The container name as the final argument can't be quoted. This happens in a 
number of places.
{code}
quote_and_append_arg(_buffer, _buffer_size, "", container_name);
{code}

See this comment for what I had found previously.
[YARN-6726 
Items|https://issues.apache.org/jira/browse/YARN-6726?focusedCommentId=16082179=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16082179]

> Add support to turn off launching privileged containers in the 
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch, 
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, 
> YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, 
> YARN-6623.009.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should 
> add a flag to the container-executor.cfg allowing admins to disable launching 
> privileged containers at the container-executor level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7193) Implement REST API for register Yarnfile in application catalog

2017-09-16 Thread Eric Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated YARN-7193:

Attachment: YARN-7193.yarn-native-services.005.patch

> Implement REST API for register Yarnfile in application catalog
> ---
>
> Key: YARN-7193
> URL: https://issues.apache.org/jira/browse/YARN-7193
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: applications
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7193.yarn-native-services.001.patch, 
> YARN-7193.yarn-native-services.002.patch, 
> YARN-7193.yarn-native-services.003.patch, 
> YARN-7193.yarn-native-services.004.patch, 
> YARN-7193.yarn-native-services.005.patch
>
>
> For support ability to register and index Yarnfile, we need a set of REST API 
> to register, search, and recommend applications from application catalog.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

2017-09-16 Thread Soumabrata Chakraborty (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169017#comment-16169017
 ] 

Soumabrata Chakraborty commented on YARN-6475:
--

[~asuresh]  [~miklos.szeg...@cloudera.com]  [~templedf]

Patch for branch-2 has been attached to this JIRA.

> Fix some long function checkstyle issues
> 
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6475.001.patch, YARN-6475-branch-2.001.patch
>
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
>   @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
>   @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Issue Comment Deleted] (YARN-6475) Fix some long function checkstyle issues

2017-09-16 Thread Soumabrata Chakraborty (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Soumabrata Chakraborty updated YARN-6475:
-
Comment: was deleted

(was: Patch for branch-2)

> Fix some long function checkstyle issues
> 
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6475.001.patch, YARN-6475-branch-2.001.patch
>
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
>   @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
>   @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6475) Fix some long function checkstyle issues

2017-09-16 Thread Soumabrata Chakraborty (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Soumabrata Chakraborty updated YARN-6475:
-
Attachment: YARN-6475-branch-2.001.patch

Patch for branch-2

> Fix some long function checkstyle issues
> 
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6475.001.patch, YARN-6475-branch-2.001.patch
>
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
>   @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
>   @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

2017-09-16 Thread Soumabrata Chakraborty (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169015#comment-16169015
 ] 

Soumabrata Chakraborty commented on YARN-6475:
--

Please ignore https://github.com/apache/hadoop/pull/274
I had created that by mistake - I have closed that PR

> Fix some long function checkstyle issues
> 
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6475.001.patch
>
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
>   @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
>   @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

2017-09-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169013#comment-16169013
 ] 

ASF GitHub Bot commented on YARN-6475:
--

GitHub user soumabrata-chakraborty opened a pull request:

https://github.com/apache/hadoop/pull/275

YARN-6475: Refactor long methods in hadoop-yarn-server-nodemanager

Refactor all methods in hadoop-yarn-server-nodemanager that exceed 150 
lines in length. Also fixes the method length related Checkstyle violations.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/soumabrata-chakraborty/hadoop 
YARN-6475-branch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/275.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #275


commit 0e15398699abef9c037aef51f860d2c2fc1f
Author: Soumabrata Chakraborty 
Date:   2017-09-16T18:28:02Z

Refactor code for long methods (above 150 lines in length)




> Fix some long function checkstyle issues
> 
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6475.001.patch
>
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
>   @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
>   @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

2017-09-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169010#comment-16169010
 ] 

ASF GitHub Bot commented on YARN-6475:
--

Github user soumabrata-chakraborty commented on the issue:

https://github.com/apache/hadoop/pull/274
  
Created by mistake! Please ignore!


> Fix some long function checkstyle issues
> 
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6475.001.patch
>
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
>   @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
>   @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

2017-09-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169011#comment-16169011
 ] 

ASF GitHub Bot commented on YARN-6475:
--

Github user soumabrata-chakraborty closed the pull request at:

https://github.com/apache/hadoop/pull/274


> Fix some long function checkstyle issues
> 
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6475.001.patch
>
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
>   @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
>   @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

2017-09-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169009#comment-16169009
 ] 

ASF GitHub Bot commented on YARN-6475:
--

GitHub user soumabrata-chakraborty opened a pull request:

https://github.com/apache/hadoop/pull/274

YARN-6475: Refactor long methods in hadoop-yarn-server-nodemanager

Refactor all methods in hadoop-yarn-server-nodemanager that exceed 150 
lines in length. Also fixes the method length related Checkstyle violations.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/soumabrata-chakraborty/hadoop branch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/274.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #274


commit bfdd4cf1e88a6a6e765a7d79297c4d4f288648b4
Author: Soumabrata Chakraborty 
Date:   2017-09-16T18:07:48Z

Refactor code for long methods (above 150 lines in length)




> Fix some long function checkstyle issues
> 
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6475.001.patch
>
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
>   @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
>   @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7149) Cross-queue preemption sometimes starves an underserved queue

2017-09-16 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168878#comment-16168878
 ] 

Hudson commented on YARN-7149:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12892 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/12892/])
YARN-7149. Cross-queue preemption sometimes starves an underserved (wangda: rev 
38c14ef8d8a094a7101917eb77d90f5e62324f61)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UsersManager.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java


> Cross-queue preemption sometimes starves an underserved queue
> -
>
> Key: YARN-7149
> URL: https://issues.apache.org/jira/browse/YARN-7149
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Eric Payne
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-7149.001.patch, YARN-7149.002.patch, 
> YARN-7149.demo.unit-test.patch
>
>
> In branch 2 and trunk, I am consistently seeing some use cases where 
> cross-queue preemption does not happen when it should. I do not see this in 
> branch-2.8.
> Use Case:
> | | *Size* | *Minimum Container Size* |
> |MyCluster | 20 GB | 0.5 GB |
> | *Queue Name* | *Capacity* | *Absolute Capacity* | *Minimum User Limit 
> Percent (MULP)* | *User Limit Factor (ULF)* |
> |Q1 | 50% = 10 GB | 100% = 20 GB | 10% = 1 GB | 2.0 |
> |Q2 | 50% = 10 GB | 100% = 20 GB | 10% = 1 GB | 2.0 |
> - {{User1}} launches {{App1}} in {{Q1}} and consumes all resources (20 GB)
> - {{User2}} launches {{App2}} in {{Q2}} and requests 10 GB
> - _Note: containers are 0.5 GB._
> - Preemption monitor kills 2 containers (equals 1 GB) from {{App1}} in {{Q1}}.
> - Capacity Scheduler assigns 2 containers (equals 1 GB) to {{App2}} in {{Q2}}.
> - _No more containers are ever preempted, even though {{Q2}} is far 
> underserved_



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6977) Node information is not provided for non am containers in RM logs

2017-09-16 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168877#comment-16168877
 ] 

Hudson commented on YARN-6977:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12892 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/12892/])
YARN-6977. Node information is not provided for non am containers in RM 
(wangda: rev 958e8c0e257216c82f68fee726e5280a919da94a)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerNode.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java


> Node information is not provided for non am containers in RM logs
> -
>
> Key: YARN-6977
> URL: https://issues.apache.org/jira/browse/YARN-6977
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Reporter: Sumana Sathish
>Assignee: Suma Shivaprasad
>  Labels: newbie
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6977.002.patch, YARN-6977.patch
>
>
> There is no information on which node non am container is being assigned in 
> the trunk for hadoop 3.0
> Earlier we used to have logs for non am container in the similar way
> {code}
> Assigned container container_ of capacity  on host 
> , which has 1 containers,  used and 
>  available after allocation
> {code}
> 3.0 has information for am container alone in the following way
> {code}
> Done launching container Container: [ContainerId: container_, 
> AllocationRequestId: 0, Version: 0, NodeId:nodeID, NodeHttpAddress: 
> nodeAddress, Resource: , Priority: 0, Token: Token { 
> kind: ContainerToken, service: service}, ExecutionType: GUARANTEED, ] for AM 
> appattempt_
> {code}
> Can we please have similar message for Non am container too ??



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Reopened] (YARN-7153) Remove duplicated code in AMRMClientAsyncImpl.java

2017-09-16 Thread Sen Zhao (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sen Zhao reopened YARN-7153:


> Remove duplicated code in AMRMClientAsyncImpl.java
> --
>
> Key: YARN-7153
> URL: https://issues.apache.org/jira/browse/YARN-7153
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Sen Zhao
>Assignee: Sen Zhao
> Attachments: YARN-7153.001.patch
>
>
> I notice there are some codes in AMRMClientAsyncImpl#setHeartbeatInterval 
> duplicate that in AMRMClientAsync#setHeartbeatInterval. The two methods are 
> handled the same way. Submit a patch to fix it!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7193) Implement REST API for register Yarnfile in application catalog

2017-09-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168833#comment-16168833
 ] 

Hadoop QA commented on YARN-7193:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 13 new or modified test 
files. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
34s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
28s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 8s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  5m 
26s{color} | {color:green} yarn-native-services passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services 
hadoop-yarn-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
8s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
59s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
23s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  3m  
7s{color} | {color:red} hadoop-yarn-project in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 11s{color} | {color:orange} root: The patch generated 1 new + 0 unchanged - 
6 fixed = 1 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  6m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
 0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green}  0m 
12s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 15 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
8s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services 
hadoop-yarn-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
21s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m 12s{color} 
| {color:red} hadoop-yarn-services in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m 11s{color} 
| {color:red} hadoop-yarn-services-core in the patch

[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts

[jira] [Comment Edited] (YARN-6475) Fix some long function checkstyle issues

[jira] [Commented] (YARN-7193) Implement REST API for register Yarnfile in application catalog

[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor

[jira] [Updated] (YARN-7193) Implement REST API for register Yarnfile in application catalog

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

[jira] [Issue Comment Deleted] (YARN-6475) Fix some long function checkstyle issues

[jira] [Updated] (YARN-6475) Fix some long function checkstyle issues

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

[jira] [Commented] (YARN-7149) Cross-queue preemption sometimes starves an underserved queue

[jira] [Commented] (YARN-6977) Node information is not provided for non am containers in RM logs

[jira] [Reopened] (YARN-7153) Remove duplicated code in AMRMClientAsyncImpl.java

[jira] [Commented] (YARN-7193) Implement REST API for register Yarnfile in application catalog

17 matches

Site Navigation

Mail list logo

Footer information