[jira] [Updated] (YARN-8053) Add hadoop-distcp in exclusion in hbase-server dependencies for timelineservice-hbase packages.
[
https://issues.apache.org/jira/browse/YARN-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8053:
Summary: Add hadoop-distcp in exclusion in hbase-server dependencies for
timelineservice-hbase packages. (was: Exclude hadoop-distcp dependencies in
hbase-server for timelineservice-hbase-client package.)
> Add hadoop-distcp in exclusion in hbase-server dependencies for
> timelineservice-hbase packages.
> ---
>
> Key: YARN-8053
> URL: https://issues.apache.org/jira/browse/YARN-8053
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8053-YARN-7055.01.patch, YARN-8053.01.patch
>
>
> It is observed that when we change the version number of hadoop leading build
> failure because of dependency resolution conflicts for HBase-2 compilation.
> We see below error which tells that hbase-server has dependency on
> hadoop-distcp. We also need to exclude hadoop-distcp from exclusion list.
> {code}
> 07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
> project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.3.0-SNAPSHOT:
> Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.3.0-SNAPSHOT
> in public
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8053) Exclude hadoop-distcp dependencies in hbase-server for timelineservice-hbase-client package.
[
https://issues.apache.org/jira/browse/YARN-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8053:
Description:
It is observed that when we change the version number of hadoop leading build
failure because of dependency resolution conflicts for HBase-2 compilation. We
see below error which tells that hbase-server has dependency on hadoop-distcp.
We also need to exclude hadoop-distcp from exclusion list.
{code}
07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
dependencies for project
org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.3.0-SNAPSHOT:
Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.3.0-SNAPSHOT in
public
{code}
was:
It is observed that when we change the version number of hadoop leading build
failure because of dependency resolution conflicts. We see below error which
tells that hbase-server has dependency on hadoop-distcp. We also need to
exclude hadoop-distcp from exclusion list.
{code}
07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
dependencies for project
org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.3.0-SNAPSHOT:
Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.3.0-SNAPSHOT in
public
{code}
> Exclude hadoop-distcp dependencies in hbase-server for
> timelineservice-hbase-client package.
>
>
> Key: YARN-8053
> URL: https://issues.apache.org/jira/browse/YARN-8053
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8053-YARN-7055.01.patch, YARN-8053.01.patch
>
>
> It is observed that when we change the version number of hadoop leading build
> failure because of dependency resolution conflicts for HBase-2 compilation.
> We see below error which tells that hbase-server has dependency on
> hadoop-distcp. We also need to exclude hadoop-distcp from exclusion list.
> {code}
> 07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
> project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.3.0-SNAPSHOT:
> Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.3.0-SNAPSHOT
> in public
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8053) Exclude hadoop-distcp dependencies in hbase-server for timelineservice-hbase-client package.
[
https://issues.apache.org/jira/browse/YARN-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8053:
Issue Type: Sub-task (was: Bug)
Parent: YARN-7213
> Exclude hadoop-distcp dependencies in hbase-server for
> timelineservice-hbase-client package.
>
>
> Key: YARN-8053
> URL: https://issues.apache.org/jira/browse/YARN-8053
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8053-YARN-7055.01.patch, YARN-8053.01.patch
>
>
> It is observed that when we change the version number of hadoop leading build
> failure because of dependency resolution conflicts. We see below error which
> tells that hbase-server has dependency on hadoop-distcp. We also need to
> exclude hadoop-distcp from exclusion list.
> {code}
> 07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
> project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.3.0-SNAPSHOT:
> Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.3.0-SNAPSHOT
> in public
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8053) Exclude hadoop-distcp dependencies in hbase-server for timelineservice-hbase-client package.
[
https://issues.apache.org/jira/browse/YARN-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405825#comment-16405825
]
Rohith Sharma K S commented on YARN-8053:
-
[~haibo.chen] would you review the patch please?
> Exclude hadoop-distcp dependencies in hbase-server for
> timelineservice-hbase-client package.
>
>
> Key: YARN-8053
> URL: https://issues.apache.org/jira/browse/YARN-8053
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8053-YARN-7055.01.patch, YARN-8053.01.patch
>
>
> It is observed that when we change the version number of hadoop leading build
> failure because of dependency resolution conflicts. We see below error which
> tells that hbase-server has dependency on hadoop-distcp. We also need to
> exclude hadoop-distcp from exclusion list.
> {code}
> 07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
> project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.3.0-SNAPSHOT:
> Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.3.0-SNAPSHOT
> in public
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8053) Exclude hadoop-distcp dependencies in hbase-server for timelineservice-hbase-client package.
[
https://issues.apache.org/jira/browse/YARN-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8053:
Attachment: YARN-8053-YARN-7055.01.patch
> Exclude hadoop-distcp dependencies in hbase-server for
> timelineservice-hbase-client package.
>
>
> Key: YARN-8053
> URL: https://issues.apache.org/jira/browse/YARN-8053
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8053-YARN-7055.01.patch, YARN-8053.01.patch
>
>
> It is observed that when we change the version number of hadoop leading build
> failure because of dependency resolution conflicts. We see below error which
> tells that hbase-server has dependency on hadoop-distcp. We also need to
> exclude hadoop-distcp from exclusion list.
> {code}
> 07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
> project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.3.0-SNAPSHOT:
> Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.3.0-SNAPSHOT
> in public
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8053) Exclude hadoop-distcp dependencies in hbase-server for timelineservice-hbase-client package.
[
https://issues.apache.org/jira/browse/YARN-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8053:
Attachment: YARN-8053.01.patch
> Exclude hadoop-distcp dependencies in hbase-server for
> timelineservice-hbase-client package.
>
>
> Key: YARN-8053
> URL: https://issues.apache.org/jira/browse/YARN-8053
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8053.01.patch
>
>
> It is observed that when we change the version number of hadoop leading build
> failure because of dependency resolution conflicts. We see below error which
> tells that hbase-server has dependency on hadoop-distcp. We also need to
> exclude hadoop-distcp from exclusion list.
> {code}
> 07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
> project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.3.0-SNAPSHOT:
> Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.3.0-SNAPSHOT
> in public
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8053) Exclude hadoop-distcp dependencies in hbase-server for timelineservice-hbase-client package.
[
https://issues.apache.org/jira/browse/YARN-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405818#comment-16405818
]
Rohith Sharma K S commented on YARN-8053:
-
To reproduce the issue, follow the below steps
# Update trunk version to 3.3-snapshot using command *mvn versions:set
-DnewVersion=3.3.0-SNAPSHOT*
# Build hadoop with -Dhbase.profile=2.0, we see the error.
But in community it is likely to happen this error since default compilation is
hbase-1.2.6 which keeps hadoop-dist jar in repository. Later on this will be
cached and used.
> Exclude hadoop-distcp dependencies in hbase-server for
> timelineservice-hbase-client package.
>
>
> Key: YARN-8053
> URL: https://issues.apache.org/jira/browse/YARN-8053
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
>
> It is observed that when we change the version number of hadoop leading build
> failure because of dependency resolution conflicts. We see below error which
> tells that hbase-server has dependency on hadoop-distcp. We also need to
> exclude hadoop-distcp from exclusion list.
> {code}
> 07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
> project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.3.0-SNAPSHOT:
> Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.3.0-SNAPSHOT
> in public
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8053) Exclude hadoop-distcp dependencies in hbase-server for timelineservice-hbase-client package.
[
https://issues.apache.org/jira/browse/YARN-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8053:
Description:
It is observed that when we change the version number of hadoop leading build
failure because of dependency resolution conflicts. We see below error which
tells that hbase-server has dependency on hadoop-distcp. We also need to
exclude hadoop-distcp from exclusion list.
{code}
07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
dependencies for project
org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.3.0-SNAPSHOT:
Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.3.0-SNAPSHOT in
public
{code}
was:
It is observed that when we change the version number of hadoop leading build
failure because of dependency resolution conflicts. We see below error which
tells that hbase-server has dependency on hadoop-distcp. We also need to
exclude hadoop-distcp from exclusion list.
{code}
07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
dependencies for project
org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.0.0.3.0.0.0-1059:
Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.0.0.3.0.0.0-1059
in public
{code}
> Exclude hadoop-distcp dependencies in hbase-server for
> timelineservice-hbase-client package.
>
>
> Key: YARN-8053
> URL: https://issues.apache.org/jira/browse/YARN-8053
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
>
> It is observed that when we change the version number of hadoop leading build
> failure because of dependency resolution conflicts. We see below error which
> tells that hbase-server has dependency on hadoop-distcp. We also need to
> exclude hadoop-distcp from exclusion list.
> {code}
> 07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
> project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.3.0-SNAPSHOT:
> Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.3.0-SNAPSHOT
> in public
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7905) Parent directory permission incorrect during public localization
[ https://issues.apache.org/jira/browse/YARN-7905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405810#comment-16405810 ] Bibin A Chundatt commented on YARN-7905: +1 for latest patch. Will commit patch today. > Parent directory permission incorrect during public localization > - > > Key: YARN-7905 > URL: https://issues.apache.org/jira/browse/YARN-7905 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bilwa S T >Priority: Critical > Attachments: YARN-7905-001.patch, YARN-7905-002.patch, > YARN-7905-003.patch, YARN-7905-004.patch, YARN-7905-005.patch, > YARN-7905-006.patch, YARN-7905-007.patch > > > Similar to YARN-6708 during public localization also we have to take care for > parent directory if the umask is 027 during node manager start up. > /filecache/0/200 > the directory permission of /filecache/0 is 750. Which cause > application failure -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8053) Exclude hadoop-distcp dependencies in hbase-server for timelineservice-hbase-client package.
[
https://issues.apache.org/jira/browse/YARN-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405804#comment-16405804
]
Rohith Sharma K S commented on YARN-8053:
-
The below dependency tree comparison shows that hbase-2 added dependency for
hadoop-dist.
* hbase-server:2.0.0-beta-1 introduced hadoop-distcp dependency which is
causing build failure.
{code}
[INFO] +- org.apache.hbase:hbase-server:jar:2.0.0-beta-1:provided
[INFO] | +- org.apache.hbase:hbase-http:jar:2.0.0-beta-1:provided
[INFO] | | +- org.eclipse.jetty:jetty-util-ajax:jar:9.3.19.v20170502:provided
[INFO] | | +- org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
[INFO] | | | +- org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
[INFO] | | | | +-
org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.25.1:provided
[INFO] | | | | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
[INFO] | | | +- org.glassfish.jersey.core:jersey-client:jar:2.25.1:provided
[INFO] | | | +-
org.glassfish.jersey.media:jersey-media-jaxb:jar:2.25.1:provided
[INFO] | | | +- javax.annotation:javax.annotation-api:jar:1.2:provided
[INFO] | | | +- org.glassfish.hk2:hk2-api:jar:2.5.0-b32:provided
[INFO] | | | | +- org.glassfish.hk2:hk2-utils:jar:2.5.0-b32:provided
[INFO] | | | | \-
org.glassfish.hk2.external:aopalliance-repackaged:jar:2.5.0-b32:provided
[INFO] | | | +-
org.glassfish.hk2.external:javax.inject:jar:2.5.0-b32:provided
[INFO] | | | +- org.glassfish.hk2:hk2-locator:jar:2.5.0-b32:provided
[INFO] | | | | \- org.javassist:javassist:jar:3.20.0-GA:provided
[INFO] | | | \- javax.validation:validation-api:jar:1.1.0.Final:provided
[INFO] | | \-
org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.25.1:provided
[INFO] | +- org.apache.hbase:hbase-procedure:jar:2.0.0-beta-1:provided
[INFO] | | \- org.apache.hbase:hbase-common:jar:tests:2.0.0-beta-1:test
[INFO] | +- org.apache.hbase:hbase-zookeeper:jar:2.0.0-beta-1:provided
[INFO] | +- org.apache.hbase:hbase-replication:jar:2.0.0-beta-1:provided
[INFO] | +- org.apache.hbase:hbase-metrics-api:jar:2.0.0-beta-1:compile
[INFO] | +- org.apache.hbase:hbase-metrics:jar:2.0.0-beta-1:compile
[INFO] | +- org.glassfish.web:javax.servlet.jsp:jar:2.3.2:provided
[INFO] | | +- org.glassfish:javax.el:jar:3.0.1-b10:provided (version selected
from constraint [3.0.0,))
[INFO] | | \- javax.servlet.jsp:javax.servlet.jsp-api:jar:2.3.1:provided
[INFO] | +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] | +- org.jamon:jamon-runtime:jar:2.4.1:provided
[INFO] | +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
[INFO] | +- org.apache.htrace:htrace-core:jar:3.2.0-incubating:provided
[INFO] | +- com.lmax:disruptor:jar:3.3.6:provided
[INFO] | \- org.apache.hadoop:hadoop-distcp:jar:3.2.0-SNAPSHOT:provided
[INFO] \- junit:junit:jar:4.11:test
[INFO] \- org.hamcrest:hamcrest-core:jar:1.3:test\
{code}
* hbase-server:1.2.6 doesn't have hadoop-distcp dependencies.
{code}
[INFO] +- org.apache.hbase:hbase-server:jar:1.2.6:provided
[INFO] | +- org.apache.hbase:hbase-procedure:jar:1.2.6:provided
[INFO] | | \- org.apache.hbase:hbase-common:jar:tests:1.2.6:test
[INFO] | +- org.apache.hbase:hbase-prefix-tree:jar:1.2.6:provided
[INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:provided
[INFO] | +- org.apache.hbase:hbase-hadoop-compat:jar:1.2.6:provided
[INFO] | +- org.apache.hbase:hbase-hadoop2-compat:jar:1.2.6:provided
[INFO] | +- org.apache.commons:commons-math:jar:2.2:provided
[INFO] | +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:provided
[INFO] | +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:provided
[INFO] | +- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:provided
[INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
[INFO] | +- tomcat:jasper-compiler:jar:5.5.23:provided
[INFO] | +- tomcat:jasper-runtime:jar:5.5.23:provided
[INFO] | | \- commons-el:commons-el:jar:1.0:provided
[INFO] | +- org.jamon:jamon-runtime:jar:2.4.1:provided
[INFO] | \- com.lmax:disruptor:jar:3.3.0:provided
[INFO] \- junit:junit:jar:4.11:test
[INFO]\- org.hamcrest:hamcrest-core:jar:1.3:test
{code}
> Exclude hadoop-distcp dependencies in hbase-server for
> timelineservice-hbase-client package.
>
>
> Key: YARN-8053
> URL: https://issues.apache.org/jira/browse/YARN-8053
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
>
> It is observed that when we change the version number of hadoop leading build
> failure because of dependency resolution conflicts. We see below error which
> tells that hbase-server has dependency on hadoop-distcp.
[jira] [Created] (YARN-8053) Exclude hadoop-distcp dependencies in hbase-server for timelineservice-hbase-client package.
Rohith Sharma K S created YARN-8053:
---
Summary: Exclude hadoop-distcp dependencies in hbase-server for
timelineservice-hbase-client package.
Key: YARN-8053
URL: https://issues.apache.org/jira/browse/YARN-8053
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
Assignee: Rohith Sharma K S
It is observed that when we change the version number of hadoop leading build
failure because of dependency resolution conflicts. We see below error which
tells that hbase-server has dependency on hadoop-distcp. We also need to
exclude hadoop-distcp from exclusion list.
{code}
07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
dependencies for project
org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.0.0.3.0.0.0-1059:
Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.0.0.3.0.0.0-1059
in public
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7581) HBase filters are not constructed correctly in ATSv2
[ https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405793#comment-16405793 ] Rohith Sharma K S commented on YARN-7581: - [~vrushalic] Go ahead and commit it. thanks. > HBase filters are not constructed correctly in ATSv2 > > > Key: YARN-7581 > URL: https://issues.apache.org/jira/browse/YARN-7581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-7581-YARN-7055.04.patch, YARN-7581.00.patch, > YARN-7581.01.patch, YARN-7581.02.patch, YARN-7581.03.patch, > YARN-7581.04.patch, YARN-7581.05.patch > > > Post YARN-7346, > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters() and > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters() > start to fail when hbase.profile is set to 2.0) > *Error Message* > [ERROR] Failures: > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1266 > expected:<2> but was:<0> > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1523 > expected:<1> but was:<0> -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8018) Yarn service: Add support for initiating service upgrade
[
https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405729#comment-16405729
]
genericqa commented on YARN-8018:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 4s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 22 new + 150 unchanged - 3 fixed = 172 total (was 153) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 47s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 27m
59s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
5s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | YARN-8018 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915227/YARN-8018.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 3d9c44bea041 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build
[jira] [Comment Edited] (YARN-6125) The application attempt's diagnostic message should have a maximum size
[ https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405721#comment-16405721 ] Leo Chen edited comment on YARN-6125 at 3/20/18 2:46 AM: - Hi, Andras Piros, Do u know how to reproduce this scenario by test case: RM send diagnostic messages more than 1M size to ZK, then so many jobs will be hang at this moment, then ARM and SRM will switch sate all time. was (Author: leo chen): Hi, Andras Piros, Do u know how to reproduce this scenario by test case: RM send diagnostic messages more than 1M size to ZK, then so many job will be hang at this moment, then ARM and SRM will switch sate all time. > The application attempt's diagnostic message should have a maximum size > --- > > Key: YARN-6125 > URL: https://issues.apache.org/jira/browse/YARN-6125 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Daniel Templeton >Assignee: Andras Piros >Priority: Critical > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: YARN-6125.000.patch, YARN-6125.001.patch, > YARN-6125.002.patch, YARN-6125.003.patch, YARN-6125.004.patch, > YARN-6125.005.patch, YARN-6125.006.patch, YARN-6125.007.patch, > YARN-6125.008.patch, YARN-6125.009.patch > > > We've found through experience that the diagnostic message can grow > unbounded. I've seen attempts that have diagnostic messages over 1MB. Since > the message is stored in the state store, it's a bad idea to allow the > message to grow unbounded. Instead, there should be a property that sets a > maximum size on the message. > I suspect that some of the ZK state store issues we've seen in the past were > due to the size of the diagnostic messages and not to the size of the > classpath, as is the current prevailing opinion. > An open question is how best to prune the message once it grows too large. > Should we > # truncate the tail, > # truncate the head, > # truncate the middle, > # add another property to make the behavior selectable, or > # none of the above? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6125) The application attempt's diagnostic message should have a maximum size
[ https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405721#comment-16405721 ] Leo Chen edited comment on YARN-6125 at 3/20/18 2:45 AM: - Hi, Andras Piros, Do u know how to reproduce this scenario by test case: RM send diagnostic messages more than 1M size to ZK, then so many job will be hang at this moment, then ARM and SRM will switch sate all time. was (Author: leo chen): Hi, Andras Piros, Do u know how to reproduce this secenario by test case: RM send diagnostic messages more than 1M size to ZK, then so many job will be hang at this moment, then ARM and SRM will switch sate all time. > The application attempt's diagnostic message should have a maximum size > --- > > Key: YARN-6125 > URL: https://issues.apache.org/jira/browse/YARN-6125 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Daniel Templeton >Assignee: Andras Piros >Priority: Critical > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: YARN-6125.000.patch, YARN-6125.001.patch, > YARN-6125.002.patch, YARN-6125.003.patch, YARN-6125.004.patch, > YARN-6125.005.patch, YARN-6125.006.patch, YARN-6125.007.patch, > YARN-6125.008.patch, YARN-6125.009.patch > > > We've found through experience that the diagnostic message can grow > unbounded. I've seen attempts that have diagnostic messages over 1MB. Since > the message is stored in the state store, it's a bad idea to allow the > message to grow unbounded. Instead, there should be a property that sets a > maximum size on the message. > I suspect that some of the ZK state store issues we've seen in the past were > due to the size of the diagnostic messages and not to the size of the > classpath, as is the current prevailing opinion. > An open question is how best to prune the message once it grows too large. > Should we > # truncate the tail, > # truncate the head, > # truncate the middle, > # add another property to make the behavior selectable, or > # none of the above? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6125) The application attempt's diagnostic message should have a maximum size
[ https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405721#comment-16405721 ] Leo Chen commented on YARN-6125: Hi, Andras Piros, Do u know how to reproduce this secenario by test case: RM send diagnostic messages more than 1M size to ZK, then so many job will be hang at this moment, then ARM and SRM will switch sate all time. > The application attempt's diagnostic message should have a maximum size > --- > > Key: YARN-6125 > URL: https://issues.apache.org/jira/browse/YARN-6125 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Daniel Templeton >Assignee: Andras Piros >Priority: Critical > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: YARN-6125.000.patch, YARN-6125.001.patch, > YARN-6125.002.patch, YARN-6125.003.patch, YARN-6125.004.patch, > YARN-6125.005.patch, YARN-6125.006.patch, YARN-6125.007.patch, > YARN-6125.008.patch, YARN-6125.009.patch > > > We've found through experience that the diagnostic message can grow > unbounded. I've seen attempts that have diagnostic messages over 1MB. Since > the message is stored in the state store, it's a bad idea to allow the > message to grow unbounded. Instead, there should be a property that sets a > maximum size on the message. > I suspect that some of the ZK state store issues we've seen in the past were > due to the size of the diagnostic messages and not to the size of the > classpath, as is the current prevailing opinion. > An open question is how best to prune the message once it grows too large. > Should we > # truncate the tail, > # truncate the head, > # truncate the middle, > # add another property to make the behavior selectable, or > # none of the above? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8051) TestRMEmbeddedElector#testCallbackSynchronization is flakey
[
https://issues.apache.org/jira/browse/YARN-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405714#comment-16405714
]
genericqa commented on YARN-8051:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
43s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 12s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 64m
44s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | YARN-8051 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915215/YARN-8051.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml findbugs
[jira] [Commented] (YARN-7516) Security check for trusted docker image
[ https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405675#comment-16405675 ] Eric Yang commented on YARN-7516: - [~leftnoteasy] It was called trusted-registries before, but it was renamed during development. If we want to change back to trusted registries, please open a separate JIRA for discussion. Thanks > Security check for trusted docker image > --- > > Key: YARN-7516 > URL: https://issues.apache.org/jira/browse/YARN-7516 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-7516.001.patch, YARN-7516.002.patch, > YARN-7516.003.patch, YARN-7516.004.patch, YARN-7516.005.patch, > YARN-7516.006.patch, YARN-7516.007.patch, YARN-7516.008.patch, > YARN-7516.009.patch, YARN-7516.010.patch, YARN-7516.011.patch, > YARN-7516.012.patch, YARN-7516.013.patch, YARN-7516.014.patch, > YARN-7516.015.patch, YARN-7516.016.patch, YARN-7516.017.patch, > YARN-7516.018.patch > > > Hadoop YARN Services can support using private docker registry image or > docker image from docker hub. In current implementation, Hadoop security is > enforced through username and group membership, and enforce uid:gid > consistency in docker container and distributed file system. There is cloud > use case for having ability to run untrusted docker image on the same cluster > for testing. > The basic requirement for untrusted container is to ensure all kernel and > root privileges are dropped, and there is no interaction with distributed > file system to avoid contamination. We can probably enforce detection of > untrusted docker image by checking the following: > # If docker image is from public docker hub repository, the container is > automatically flagged as insecure, and disk volume mount are disabled > automatically, and drop all kernel capabilities. > # If docker image is from private repository in docker hub, and there is a > white list to allow the private repository, disk volume mount is allowed, > kernel capabilities follows the allowed list. > # If docker image is from private trusted registry with image name like > "private.registry.local:5000/centos", and white list allows this private > trusted repository. Disk volume mount is allowed, kernel capabilities > follows the allowed list. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8018) Yarn service: Add support for initiating service upgrade
[ https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405670#comment-16405670 ] Chandni Singh commented on YARN-8018: - Created YARN-8052 to address overwrite of definition during flex. > Yarn service: Add support for initiating service upgrade > > > Key: YARN-8018 > URL: https://issues.apache.org/jira/browse/YARN-8018 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8018.001.patch > > > Add support for initiating service upgrade which includes the following main > changes: > # Service API to initiate upgrade > # Persist service version on hdfs > # Start the upgraded version of service -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8018) Yarn service: Add support for initiating service upgrade
[ https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-8018: Attachment: YARN-8018.001.patch > Yarn service: Add support for initiating service upgrade > > > Key: YARN-8018 > URL: https://issues.apache.org/jira/browse/YARN-8018 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8018.001.patch > > > Add support for initiating service upgrade which includes the following main > changes: > # Service API to initiate upgrade > # Persist service version on hdfs > # Start the upgraded version of service -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8052) Move overwriting of service definition during flex to service master
Chandni Singh created YARN-8052: --- Summary: Move overwriting of service definition during flex to service master Key: YARN-8052 URL: https://issues.apache.org/jira/browse/YARN-8052 Project: Hadoop YARN Issue Type: Sub-task Reporter: Chandni Singh Assignee: Chandni Singh The overwrite of service definition during flex is done from the ServiceClient. During auto finalization of upgrade, the current service definition gets overwritten as well by the service master. This creates a potential conflict. Need to move the overwrite of service definition during flex to the ServiceClient. Discussed on YARN-8018. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7516) Security check for trusted docker image
[
https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405630#comment-16405630
]
Wangda Tan commented on YARN-7516:
--
Thanks [~eyang],
It makes sense, however I think the config name is a bit confusing:
bq. docker.privileged-containers.registries
Sounds like it only takes effect when {{--privileged}} appears. Same as the
documentation:
bq. Comma separated list of trusted docker registries for running trusted
privileged docker containers. By default, no registries are defined.
Is it better to rename it to {{docker.dontainer.trusted-registries}}?
> Security check for trusted docker image
> ---
>
> Key: YARN-7516
> URL: https://issues.apache.org/jira/browse/YARN-7516
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7516.001.patch, YARN-7516.002.patch,
> YARN-7516.003.patch, YARN-7516.004.patch, YARN-7516.005.patch,
> YARN-7516.006.patch, YARN-7516.007.patch, YARN-7516.008.patch,
> YARN-7516.009.patch, YARN-7516.010.patch, YARN-7516.011.patch,
> YARN-7516.012.patch, YARN-7516.013.patch, YARN-7516.014.patch,
> YARN-7516.015.patch, YARN-7516.016.patch, YARN-7516.017.patch,
> YARN-7516.018.patch
>
>
> Hadoop YARN Services can support using private docker registry image or
> docker image from docker hub. In current implementation, Hadoop security is
> enforced through username and group membership, and enforce uid:gid
> consistency in docker container and distributed file system. There is cloud
> use case for having ability to run untrusted docker image on the same cluster
> for testing.
> The basic requirement for untrusted container is to ensure all kernel and
> root privileges are dropped, and there is no interaction with distributed
> file system to avoid contamination. We can probably enforce detection of
> untrusted docker image by checking the following:
> # If docker image is from public docker hub repository, the container is
> automatically flagged as insecure, and disk volume mount are disabled
> automatically, and drop all kernel capabilities.
> # If docker image is from private repository in docker hub, and there is a
> white list to allow the private repository, disk volume mount is allowed,
> kernel capabilities follows the allowed list.
> # If docker image is from private trusted registry with image name like
> "private.registry.local:5000/centos", and white list allows this private
> trusted repository. Disk volume mount is allowed, kernel capabilities
> follows the allowed list.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8027) Setting hostname of docker container breaks for --net=host in docker 1.13
[ https://issues.apache.org/jira/browse/YARN-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405631#comment-16405631 ] Hudson commented on YARN-8027: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13856 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13856/]) YARN-8027. Setting hostname of docker container breaks for --net=host in (jlowe: rev f480367af68e06ed17b8018092c9987a13bb9f63) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java > Setting hostname of docker container breaks for --net=host in docker 1.13 > - > > Key: YARN-8027 > URL: https://issues.apache.org/jira/browse/YARN-8027 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-8027.001.patch > > > In DockerLinuxContainerRuntime:launchContainer, we are adding the --hostname > argument to the docker run command to set the hostname in the container to > something like: ctr-e84-1520889172376-0001-01-01. > This does not work when combined with the --net=host command line option in > Docker 1.13.1. It causes multiple failures when the client tries to resolve > the hostname and it fails. > We haven't seen this before because we were using docker 1.12.6 which seems > to ignore --hostname when you are using --net=host. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405625#comment-16405625
]
Eric Yang commented on YARN-7654:
-
Patch 004 addresses the following:
- Ensure container environment variables are passed through from user defined
values to docker without expansion.
- Instead of constructing shell command using char*, use string array for execv.
- Removed exposure of WORK_DIR variables.
- Added retry logic for running docker inspect command for acquiring pid from
docker container.
Let me know if there is any question or concerns. Thanks
> Support ENTRY_POINT for docker container
>
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-7654.001.patch, YARN-7654.002.patch,
> YARN-7654.003.patch, YARN-7654.004.patch
>
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7654:
Attachment: YARN-7654.004.patch
> Support ENTRY_POINT for docker container
>
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-7654.001.patch, YARN-7654.002.patch,
> YARN-7654.003.patch, YARN-7654.004.patch
>
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8018) Yarn service: Add support for initiating service upgrade
[ https://issues.apache.org/jira/browse/YARN-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-8018: Attachment: (was: YARN-8018.wip.patch) > Yarn service: Add support for initiating service upgrade > > > Key: YARN-8018 > URL: https://issues.apache.org/jira/browse/YARN-8018 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > > Add support for initiating service upgrade which includes the following main > changes: > # Service API to initiate upgrade > # Persist service version on hdfs > # Start the upgraded version of service -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8051) TestRMEmbeddedElector#testCallbackSynchronization is flakey
[
https://issues.apache.org/jira/browse/YARN-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated YARN-8051:
Attachment: YARN-8051.001.patch
> TestRMEmbeddedElector#testCallbackSynchronization is flakey
> ---
>
> Key: YARN-8051
> URL: https://issues.apache.org/jira/browse/YARN-8051
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: test
>Affects Versions: 3.2.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8051.001.patch
>
>
> We've seen some rare flakey failures in
> {{TestRMEmbeddedElector#testCallbackSynchronization}}:
> {noformat}
> org.mockito.exceptions.verification.WantedButNotInvoked:
> Wanted but not invoked:
> adminService.transitionToStandby();
> -> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronizationNeutral(TestRMEmbeddedElector.java:215)
> Actually, there were zero interactions with this mock.
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronizationNeutral(TestRMEmbeddedElector.java:215)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronization(TestRMEmbeddedElector.java:146)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronization(TestRMEmbeddedElector.java:109)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8051) TestRMEmbeddedElector#testCallbackSynchronization is flakey
[
https://issues.apache.org/jira/browse/YARN-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405564#comment-16405564
]
Robert Kanter commented on YARN-8051:
-
The problem is a timing issue: we're using a short {{Thread.sleep}} in the test
(and some related ones). The patch fixes this by replacing the {{Thread.sleep}}
calls with {{GenericTestUtils.waitFor}} calls. In order to get the actual count
of the numbers of calls to do this, I had to update Mockito so I could use
{{mockingDetails}}. I also replaced some silly pairs of {{verify}} statements
where we had {{atLeast(1)}} and {{atMost(1)}}, with just {{times(1)}}.
> TestRMEmbeddedElector#testCallbackSynchronization is flakey
> ---
>
> Key: YARN-8051
> URL: https://issues.apache.org/jira/browse/YARN-8051
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: test
>Affects Versions: 3.2.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8051.001.patch
>
>
> We've seen some rare flakey failures in
> {{TestRMEmbeddedElector#testCallbackSynchronization}}:
> {noformat}
> org.mockito.exceptions.verification.WantedButNotInvoked:
> Wanted but not invoked:
> adminService.transitionToStandby();
> -> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronizationNeutral(TestRMEmbeddedElector.java:215)
> Actually, there were zero interactions with this mock.
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronizationNeutral(TestRMEmbeddedElector.java:215)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronization(TestRMEmbeddedElector.java:146)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronization(TestRMEmbeddedElector.java:109)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8051) TestRMEmbeddedElector#testCallbackSynchronization is flakey
Robert Kanter created YARN-8051:
---
Summary: TestRMEmbeddedElector#testCallbackSynchronization is
flakey
Key: YARN-8051
URL: https://issues.apache.org/jira/browse/YARN-8051
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Affects Versions: 3.2.0
Reporter: Robert Kanter
Assignee: Robert Kanter
We've seen some rare flakey failures in
{{TestRMEmbeddedElector#testCallbackSynchronization}}:
{noformat}
org.mockito.exceptions.verification.WantedButNotInvoked:
Wanted but not invoked:
adminService.transitionToStandby();
-> at
org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronizationNeutral(TestRMEmbeddedElector.java:215)
Actually, there were zero interactions with this mock.
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronizationNeutral(TestRMEmbeddedElector.java:215)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronization(TestRMEmbeddedElector.java:146)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronization(TestRMEmbeddedElector.java:109)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8043) Add the exception message for failed launches running under LCE
[
https://issues.apache.org/jira/browse/YARN-8043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405557#comment-16405557
]
Eric Badger commented on YARN-8043:
---
+1 (non-binding) lgtm
> Add the exception message for failed launches running under LCE
> ---
>
> Key: YARN-8043
> URL: https://issues.apache.org/jira/browse/YARN-8043
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
>Priority: Major
> Attachments: YARN-8043.001.patch
>
>
> If a {{ContainerExecutionException}} is thrown during container launch under
> LCE information is added to the diagnostics for the container regarding the
> failure. The diagnostic's output includes the container id, exit code, "error
> output" and "shell output", with the expectation that the relevant exception
> details are in the "error output".
> {{ContainerExecutionException}} has several constructors, with the most
> commonly used being the one that accepts a String. This form does not put the
> exception details into the "error output". This makes it difficult for the
> user to troubleshoot the failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4022) queue not remove from webpage(/cluster/scheduler) when delete queue in xxx-scheduler.xml
[
https://issues.apache.org/jira/browse/YARN-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-4022:
---
Attachment: Queue Deletion in Fair Scheduler.pdf
> queue not remove from webpage(/cluster/scheduler) when delete queue in
> xxx-scheduler.xml
>
>
> Key: YARN-4022
> URL: https://issues.apache.org/jira/browse/YARN-4022
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, resourcemanager
>Affects Versions: 2.7.1
>Reporter: forrestchen
>Assignee: Szilard Nemeth
>Priority: Major
> Labels: oct16-medium, scheduler
> Attachments: Queue Deletion in Fair Scheduler.pdf,
> YARN-4022.001.patch, YARN-4022.002.patch, YARN-4022.003.patch,
> YARN-4022.004.patch
>
>
> When I delete an existing queue by modify the xxx-schedule.xml, I can still
> see the queue information block in webpage(/cluster/scheduler) though the
> 'Min Resources' items all become to zero and have no item of 'Max Running
> Applications'.
> I can still submit an application to the deleted queue and the application
> will run using 'root.default' queue instead, but submit to an un-exist queue
> will cause an exception.
> My expectation is the deleted queue will not displayed in webpage and submit
> application to the deleted queue will act just like the queue doesn't exist.
> PS: There's no application running in the queue I delete.
> Some related config in yarn-site.xml:
> {code}
>
> yarn.scheduler.fair.user-as-default-queue
> false
>
>
> yarn.scheduler.fair.allow-undeclared-pools
> false
>
> {code}
> a related question is here:
> http://stackoverflow.com/questions/26488564/hadoop-yarn-why-the-queue-cannot-be-deleted-after-i-revise-my-fair-scheduler-xm
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4022) queue not remove from webpage(/cluster/scheduler) when delete queue in xxx-scheduler.xml
[
https://issues.apache.org/jira/browse/YARN-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405546#comment-16405546
]
Yufei Gu commented on YARN-4022:
Attached the design document.
> queue not remove from webpage(/cluster/scheduler) when delete queue in
> xxx-scheduler.xml
>
>
> Key: YARN-4022
> URL: https://issues.apache.org/jira/browse/YARN-4022
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, resourcemanager
>Affects Versions: 2.7.1
>Reporter: forrestchen
>Assignee: Szilard Nemeth
>Priority: Major
> Labels: oct16-medium, scheduler
> Attachments: Queue Deletion in Fair Scheduler.pdf,
> YARN-4022.001.patch, YARN-4022.002.patch, YARN-4022.003.patch,
> YARN-4022.004.patch
>
>
> When I delete an existing queue by modify the xxx-schedule.xml, I can still
> see the queue information block in webpage(/cluster/scheduler) though the
> 'Min Resources' items all become to zero and have no item of 'Max Running
> Applications'.
> I can still submit an application to the deleted queue and the application
> will run using 'root.default' queue instead, but submit to an un-exist queue
> will cause an exception.
> My expectation is the deleted queue will not displayed in webpage and submit
> application to the deleted queue will act just like the queue doesn't exist.
> PS: There's no application running in the queue I delete.
> Some related config in yarn-site.xml:
> {code}
>
> yarn.scheduler.fair.user-as-default-queue
> false
>
>
> yarn.scheduler.fair.allow-undeclared-pools
> false
>
> {code}
> a related question is here:
> http://stackoverflow.com/questions/26488564/hadoop-yarn-why-the-queue-cannot-be-deleted-after-i-revise-my-fair-scheduler-xm
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7707) [GPG] Policy generator framework
[
https://issues.apache.org/jira/browse/YARN-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405457#comment-16405457
]
genericqa commented on YARN-7707:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} YARN-7402 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
47s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
44s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
8s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
YARN-7402 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} YARN-7402 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 58s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
59s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
2s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hadoop-yarn-server-globalpolicygenerator in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7707 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915176/YARN-7707-YARN-7402.11.patch
|
| Optional Tests | asflicense compile javac javadoc
[jira] [Commented] (YARN-7221) Add security check for privileged docker container
[ https://issues.apache.org/jira/browse/YARN-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405438#comment-16405438 ] Eric Yang commented on YARN-7221: - The failed test cases are not related to this patch. > Add security check for privileged docker container > -- > > Key: YARN-7221 > URL: https://issues.apache.org/jira/browse/YARN-7221 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0, 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-7221.001.patch, YARN-7221.002.patch, > YARN-7221.003.patch, YARN-7221.004.patch, YARN-7221.005.patch, > YARN-7221.006.patch, YARN-7221.007.patch > > > When a docker is running with privileges, majority of the use case is to have > some program running with root then drop privileges to another user. i.e. > httpd to start with privileged and bind to port 80, then drop privileges to > www user. > # We should add security check for submitting users, to verify they have > "sudo" access to run privileged container. > # We should remove --user=uid:gid for privileged containers. > > Docker can be launched with --privileged=true, and --user=uid:gid flag. With > this parameter combinations, user will not have access to become root user. > All docker exec command will be drop to uid:gid user to run instead of > granting privileges. User can gain root privileges if container file system > contains files that give user extra power, but this type of image is > considered as dangerous. Non-privileged user can launch container with > special bits to acquire same level of root power. Hence, we lose control of > which image should be run with --privileges, and who have sudo rights to use > privileged container images. As the result, we should check for sudo access > then decide to parameterize --privileged=true OR --user=uid:gid. This will > avoid leading developer down the wrong path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6936) [Atsv2] Retrospect storing entities into sub application table from client perspective
[ https://issues.apache.org/jira/browse/YARN-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405289#comment-16405289 ] Vrushali C commented on YARN-6936: -- Thanks for the patch [~rohithsharma] ! I tend to agree with [~haibochen] . It will be good to provide another API to write sub app entities instead of having all entities written to sub app table. This will give applications the flexibility to store only some relevant entities to sub app table. Also, minor point. Update the sentence in the documentation from "This is usefull when entity need to be queried outside the scope of application." to "This is useful when entities need to be queried outside the scope of application." > [Atsv2] Retrospect storing entities into sub application table from client > perspective > -- > > Key: YARN-6936 > URL: https://issues.apache.org/jira/browse/YARN-6936 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-6936.000.patch > > > Currently YARN-6734 stores entities into sub application table only if doAs > user and submitted users are different. This holds good for Tez kind of use > cases. But AM runs as same as submitted user like MR also need to store > entities in sub application table so that it could read entities without > application id. > This would be a point of concern later stages when ATSv2 is deployed into > production. This JIRA is to retrospect decision of storing entities into sub > application table based on client side configuration driven rather than user > driven. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7707) [GPG] Policy generator framework
[ https://issues.apache.org/jira/browse/YARN-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-7707: - Attachment: YARN-7707-YARN-7402.11.patch > [GPG] Policy generator framework > > > Key: YARN-7707 > URL: https://issues.apache.org/jira/browse/YARN-7707 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Young Chen >Priority: Major > Labels: federation, gpg > Attachments: YARN-7707-YARN-7402.01.patch, > YARN-7707-YARN-7402.02.patch, YARN-7707-YARN-7402.03.patch, > YARN-7707-YARN-7402.04.patch, YARN-7707-YARN-7402.05.patch, > YARN-7707-YARN-7402.06.patch, YARN-7707-YARN-7402.07.patch, > YARN-7707-YARN-7402.08.patch, YARN-7707-YARN-7402.09.patch, > YARN-7707-YARN-7402.10.patch, YARN-7707-YARN-7402.11.patch > > > This JIRA tracks the development of a generic framework for querying > sub-clusters for metrics, running policies, and updating them in the > FederationStateStore. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6936) [Atsv2] Retrospect storing entities into sub application table from client perspective
[ https://issues.apache.org/jira/browse/YARN-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405268#comment-16405268 ] Haibo Chen commented on YARN-6936: -- [~rohithsharma]. Can you elaborate on why we want to expose the decision in client side configuration? It is a binary decision in this patch, but I think some frameworks may want to post both application entities and subapplication entities. What do you think if we provide two endpoints in TimelineCollectorWebServices, one for application and the other sub-application entities? > [Atsv2] Retrospect storing entities into sub application table from client > perspective > -- > > Key: YARN-6936 > URL: https://issues.apache.org/jira/browse/YARN-6936 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-6936.000.patch > > > Currently YARN-6734 stores entities into sub application table only if doAs > user and submitted users are different. This holds good for Tez kind of use > cases. But AM runs as same as submitted user like MR also need to store > entities in sub application table so that it could read entities without > application id. > This would be a point of concern later stages when ATSv2 is deployed into > production. This JIRA is to retrospect decision of storing entities into sub > application table based on client side configuration driven rather than user > driven. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7675) [UI2] Support loading pre-2.8 version /scheduler REST response for queue page
[
https://issues.apache.org/jira/browse/YARN-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7675:
-
Fix Version/s: (was: 3.2.0)
> [UI2] Support loading pre-2.8 version /scheduler REST response for queue page
> -
>
> Key: YARN-7675
> URL: https://issues.apache.org/jira/browse/YARN-7675
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Gergely Novák
>Assignee: Gergely Novák
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7675.001.patch
>
>
> If we connect the new YARN UI to any Hadoop versions older than 2.8 it won't
> load. The console shows this trace:
> {noformat}
> TypeError: Cannot read property 'queueCapacitiesByPartition' of undefined
> at Class.normalizeSingleResponse (yarn-ui.js:13903)
> at Class.superWrapper [as normalizeSingleResponse] (vendor.js:31811)
> at Class.handleQueue (yarn-ui.js:13928)
> at Class.normalizeArrayResponse (yarn-ui.js:13952)
> at Class.normalizeQueryResponse (vendor.js:101566)
> at Class.normalizeResponse (vendor.js:101468)
> at
> ember$data$lib$system$store$serializer$response$$normalizeResponseHelper
> (vendor.js:95345)
> at vendor.js:95672
> at Backburner.run (vendor.js:10426)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8024) LOG in class MaxRunningAppsEnforcer is initialized with a faulty class FairScheduler
[ https://issues.apache.org/jira/browse/YARN-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8024: - Fix Version/s: (was: 3.2.0) > LOG in class MaxRunningAppsEnforcer is initialized with a faulty class > FairScheduler > - > > Key: YARN-8024 > URL: https://issues.apache.org/jira/browse/YARN-8024 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Sen Zhao >Priority: Major > Labels: newbie++ > Fix For: 3.1.0 > > Attachments: YARN-8024.001.patch > > > It should be initialized with class MaxRunningAppsEnforcer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7346) Add a profile to allow optional compilation for ATSv2 with HBase-2.0
[ https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7346: - Fix Version/s: (was: 3.2.0) > Add a profile to allow optional compilation for ATSv2 with HBase-2.0 > > > Key: YARN-7346 > URL: https://issues.apache.org/jira/browse/YARN-7346 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Haibo Chen >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-7346.00.patch, YARN-7346.01.patch, > YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, > YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, > YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08-incremental.patch, > YARN-7346.08.patch, YARN-7346.09.patch, YARN-7346.10.patch, > YARN-7346.11.patch, YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, > YARN-7581.prelim.patch, > hadoop-yarn-server-timelineservice-hbase-server-1-findbugsXml.xml, > hadoop-yarn-server-timelineservice-hbase-server-1-javadoc-report.txt, > hadoop-yarn-server-timelineservice-hbase-server-2-findbugsXml.xml, > hadoop-yarn-server-timelineservice-hbase-server-2-javadoc-report.txt > > > When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, > I got the following errors: > [https://pastebin.com/Ms4jYEVB] > This issue is to fix the compilation errors. > The scope of the Jira is to add a profile to allow optional compilation for > ATSv2 with HBase2.0. The default compilation for trunk will still be for > hbase 1.2.6. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7929) Support to set container execution type in SLS
[ https://issues.apache.org/jira/browse/YARN-7929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7929: - Fix Version/s: (was: 3.2.0) > Support to set container execution type in SLS > -- > > Key: YARN-7929 > URL: https://issues.apache.org/jira/browse/YARN-7929 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-7929.001.patch, YARN-7929.0010.patch, > YARN-7929.002.patch, YARN-7929.003.patch, YARN-7929.004.patch, > YARN-7929.005.patch, YARN-7929.006.patch, YARN-7929.007.patch, > YARN-7929.008.patch, YARN-7929.009.patch > > > SLS currently support three tracetype, SYNTH, SLS and RUMEN, but trace file > can not set execution type of container. > This jira will introduce execution type in SLS to help better simulation. > This will help the perf testing with regarding to the Opportunistic > Containers. > RUMEN has default execution type GUARANTEED > SYNTH set execution type by field map_execution_type and > reduce_execution_type > SLS set execution type by field container.execution_type > For compatibility set GUARANTEED as default value when not setting above > fields in trace file -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7891) LogAggregationIndexedFileController should support read from HAR file
[ https://issues.apache.org/jira/browse/YARN-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7891: - Fix Version/s: (was: 3.2.0) > LogAggregationIndexedFileController should support read from HAR file > - > > Key: YARN-7891 > URL: https://issues.apache.org/jira/browse/YARN-7891 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-7891.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7446) Docker container privileged mode and --user flag contradict each other
[ https://issues.apache.org/jira/browse/YARN-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7446: - Fix Version/s: (was: 3.2.0) > Docker container privileged mode and --user flag contradict each other > -- > > Key: YARN-7446 > URL: https://issues.apache.org/jira/browse/YARN-7446 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0, 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-7446.001.patch, YARN-7446.002.patch, > YARN-7446.003.patch, YARN-7446.004.patch > > > In the current implementation, when privileged=true, --user flag is also > passed to docker for launching container. In reality, the container has no > way to use root privileges unless there is sticky bit or sudoers in the image > for the specified user to gain privileges again. To avoid duplication of > dropping and reacquire root privileges, we can reduce the duplication of > specifying both flag. When privileged mode is enabled, --user flag should be > omitted. When non-privileged mode is enabled, --user flag is supplied. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7952) RM should be able to recover log aggregation status after restart/fail-over
[ https://issues.apache.org/jira/browse/YARN-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7952: - Fix Version/s: (was: 3.2.0) > RM should be able to recover log aggregation status after restart/fail-over > --- > > Key: YARN-7952 > URL: https://issues.apache.org/jira/browse/YARN-7952 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-7952-poc.patch, YARN-7952.1.patch, > YARN-7952.2.patch, YARN-7952.3.patch, YARN-7952.3.patch, YARN-7952.5.patch, > YARN-7952.6.patch, YARN-7952.7.patch, YARN-7952.8.patch, YARN-7952.9.patch > > > Right now, the NM would send its own log aggregation status to RM > periodically to RM. And RM would aggregate the status for each application, > but it will not generate the final status until a client call(from web ui or > cli) trigger it. But RM never persists the log aggregation status. So, when > RM restarts/fails over, the log aggregation status will become “NOT_STARTED”. > This is confusing, maybe we should change it to “NOT_AVAILABLE” (will create > a separate ticket for this). Anyway, we need to persist the log aggregation > status for the future use. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7949) [UI2] ArtifactsId should not be a compulsory field for new service
[
https://issues.apache.org/jira/browse/YARN-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7949:
-
Fix Version/s: (was: 3.2.0)
> [UI2] ArtifactsId should not be a compulsory field for new service
> --
>
> Key: YARN-7949
> URL: https://issues.apache.org/jira/browse/YARN-7949
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Yesha Vora
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7949.001.patch
>
>
> 1) Click on New Service
> 2) Create a component
> Create Component page has Artifacts Id as compulsory entry. Few yarn service
> example such as sleeper.json does not need to provide artifacts id.
> {code:java|title=sleeper.json}
> {
> "name": "sleeper-service",
> "components" :
> [
> {
> "name": "sleeper",
> "number_of_containers": 2,
> "launch_command": "sleep 90",
> "resource": {
> "cpus": 1,
> "memory": "256"
> }
> }
> ]
> }{code}
> Thus, artifactsId should not be compulsory field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7835) [Atsv2] Race condition in NM while publishing events if second attempt is launched on the same node
[ https://issues.apache.org/jira/browse/YARN-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7835: - Fix Version/s: (was: 3.2.0) > [Atsv2] Race condition in NM while publishing events if second attempt is > launched on the same node > --- > > Key: YARN-7835 > URL: https://issues.apache.org/jira/browse/YARN-7835 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Critical > Fix For: 3.1.0, 2.10.0 > > Attachments: YARN-7835.001.patch, YARN-7835.002.patch, > YARN-7835.003.patch, YARN-7835.004.patch > > > It is observed race condition that if master container is killed for some > reason and launched on same node then NMTimelinePublisher doesn't add > timelineClient. But once completed container for 1st attempt has come then > NMTimelinePublisher removes the timelineClient. > It causes all subsequent event publishing from different client fails to > publish with exception Application is not found. ! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8011) TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart fails sometimes in trunk
[
https://issues.apache.org/jira/browse/YARN-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-8011:
-
Fix Version/s: (was: 3.2.0)
> TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart
> fails sometimes in trunk
> ---
>
> Key: YARN-8011
> URL: https://issues.apache.org/jira/browse/YARN-8011
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: YARN-8011.001.patch, YARN-8011.002.patch
>
>
> TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart
> often pass, but the following errors sometimes occur:
> {noformat}
> java.lang.AssertionError:
> Expected :15360
> Actual :14336
>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService.verifyMetrics(TestOpportunisticContainerAllocatorAMService.java:732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService.testContainerPromoteAndDemoteBeforeContainerStart(TestOpportunisticContainerAllocatorAMService.java:330)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
>
> This problem is caused by that deducting resource is a little behind the
> assertion. To solve this problem, It can sleep a while before this assertion
> as below.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7223) Document GPU isolation feature
[ https://issues.apache.org/jira/browse/YARN-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7223: - Fix Version/s: (was: 3.2.0) > Document GPU isolation feature > -- > > Key: YARN-7223 > URL: https://issues.apache.org/jira/browse/YARN-7223 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Fix For: 3.1.0 > > Attachments: YARN-7223.002.patch, YARN-7223.002.pdf, > YARN-7223.003.patch, YARN-7223.wip.001.patch, YARN-7223.wip.001.pdf > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7995) Remove unnecessary boxings and unboxings from PlacementConstraintParser.java
[
https://issues.apache.org/jira/browse/YARN-7995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7995:
-
Fix Version/s: (was: 3.2.0)
> Remove unnecessary boxings and unboxings from PlacementConstraintParser.java
>
>
> Key: YARN-7995
> URL: https://issues.apache.org/jira/browse/YARN-7995
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Sen Zhao
>Priority: Minor
> Labels: newbie
> Fix For: 3.1.0
>
> Attachments: YARN-7995.001.patch
>
>
> {code}
> String maxCardinalityStr = resetElements.pop();
> Integer max = toInt(maxCardinalityStr);
> String minCardinalityStr = resetElements.pop();
> Integer min = toInt(minCardinalityStr);
> {code}
> {{toInt(String)}} does not return null, so {{Integer}} should be replaced
> with {{int}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8022) ResourceManager UI cluster/app/ page fails to render
[ https://issues.apache.org/jira/browse/YARN-8022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8022: - Fix Version/s: (was: 3.2.0) > ResourceManager UI cluster/app/ page fails to render > > > Key: YARN-8022 > URL: https://issues.apache.org/jira/browse/YARN-8022 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Blocker > Fix For: 3.1.0, 3.0.1 > > Attachments: Screen Shot 2018-03-12 at 1.45.05 PM.png, > YARN-8022.001.patch, YARN-8022.002.patch > > > The page displays the message "Failed to read the attempts of the application" > > The following stack trace is observed in RM log. > org.apache.hadoop.yarn.server.webapp.AppBlock: Failed to read the attempts of > the application application_1520597233415_0002. > java.lang.NullPointerException > at org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:283) > at org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:280) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682) > at org.apache.hadoop.yarn.server.webapp.AppBlock.render(AppBlock.java:279) > at > org.apache.hadoop.yarn.server.resourcemanager.webapp.RMAppBlock.render(RMAppBlock.java:71) > at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69) > at > org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79) > at org.apache.hadoop.yarn.webapp.View.render(View.java:235) > at org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49) > at > org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl$EImp._v(HamletImpl.java:117) > at org.apache.hadoop.yarn.webapp.hamlet2.Hamlet$TD.__(Hamlet.java:848) > at > org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:71) > at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82) > at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:212) > at > org.apache.hadoop.yarn.server.resourcemanager.webapp.RmController.app(RmController.java:54) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7626) Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount
[ https://issues.apache.org/jira/browse/YARN-7626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7626: - Fix Version/s: (was: 3.2.0) > Allow regular expression matching in container-executor.cfg for devices and > named docker volumes mount > -- > > Key: YARN-7626 > URL: https://issues.apache.org/jira/browse/YARN-7626 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Zian Chen >Assignee: Zian Chen >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-7626.001.patch, YARN-7626.002.patch, > YARN-7626.003.patch, YARN-7626.004.patch, YARN-7626.005.patch, > YARN-7626.006.patch, YARN-7626.007.patch, YARN-7626.008.patch, > YARN-7626.009.patch, YARN-7626.010.patch, YARN-7626.011.patch > > > Currently when we config some of the GPU devices related fields (like ) in > container-executor.cfg, these fields are generated based on different driver > versions or GPU device names. We want to enable regular expression matching > so that user don't need to manually set up these fields when config > container-executor.cfg, -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7954) Component status stays "Ready" when yarn service is stopped
[
https://issues.apache.org/jira/browse/YARN-7954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7954:
-
Fix Version/s: (was: 3.2.0)
> Component status stays "Ready" when yarn service is stopped
> ---
>
> Key: YARN-7954
> URL: https://issues.apache.org/jira/browse/YARN-7954
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Gour Saha
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7954.001.patch, YARN-7954.002.patch
>
>
> Steps:
> 1) Launch yarn service application
> 2) Stop application
> 3) Run get status from yarn cli
> {code}
> [hdpuser@cn005 sleeper]$ yarn app -status yesha-sleeper
> WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of
> YARN_LOG_DIR.
> WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of
> YARN_LOGFILE.
> WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of
> YARN_PID_DIR.
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 18/02/16 10:54:37 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/02/16 10:54:37 INFO client.RMProxy: Connecting to ResourceManager at
> xxx/xx.xx.xx.xx:8050
> 18/02/16 10:54:37 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xx.xx.xx.xx:10200
> 18/02/16 10:54:37 INFO client.RMProxy: Connecting to ResourceManager at
> xxx/xx.xx.xx.xx:8050
> 18/02/16 10:54:37 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xx.xx.xx.xx:10200
> 18/02/16 10:54:38 INFO util.log: Logging initialized @1957ms
> {"name":"yesha-sleeper","lifetime":-1,"components":[],"configuration":{"properties":{},"env":{},"files":[]},"state":"STOPPED","quicklinks":{},"kerberos_principal":{}}
> {code}
> 4) Validate UI2 for service status
> Here, Yarn service status is marked as "finished". However, components status
> still shows Ready.
> On stopping yarn service, component status should be updated to "Stop"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7985) Service name is validated twice in ServiceClient when a service is created
[
https://issues.apache.org/jira/browse/YARN-7985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7985:
-
Fix Version/s: (was: 3.2.0)
> Service name is validated twice in ServiceClient when a service is created
> --
>
> Key: YARN-7985
> URL: https://issues.apache.org/jira/browse/YARN-7985
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Affects Versions: 3.1.0
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Trivial
> Fix For: 3.1.0
>
> Attachments: YARN-7985.001.patch
>
>
> In ServiceClient.actionCreate() :
> {code:java}
> ServiceApiUtil.validateNameFormat(serviceName, getConfig());
> ServiceApiUtil.validateAndResolveService(service, fs, getConfig());{code}
> However, the ServiceApiUtil.validateAndResolveService(...), also validates
> the service name
> {code:java}
> if (StringUtils.isEmpty(service.getName())) {
> throw new IllegalArgumentException(
> RestApiErrorMessages.ERROR_APPLICATION_NAME_INVALID);
> }
> validateNameFormat(service.getName(), conf);{code}
> Also, {{ServiceClientTest}} which is a mock client for the ApiServer, can
> perform the actual validation which is performed by the {{ServiceClient}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7934) [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos
[ https://issues.apache.org/jira/browse/YARN-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7934: - Fix Version/s: (was: 3.2.0) > [GQ] Refactor preemption calculators to allow overriding for Federation > Global Algos > > > Key: YARN-7934 > URL: https://issues.apache.org/jira/browse/YARN-7934 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-7934.v1.patch, YARN-7934.v2.patch, > YARN-7934.v3.patch, YARN-7934.v4.patch > > > This Jira tracks minimal changes in the capacity scheduler preemption > mechanics that allow for sub-classing and overriding of certain behaviors, > which we use to implement federation global algorithms, e.g., in YARN-7403. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8000) Yarn Service: component instance name shows up as component name in container record
[ https://issues.apache.org/jira/browse/YARN-8000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8000: - Fix Version/s: (was: 3.2.0) > Yarn Service: component instance name shows up as component name in container > record > - > > Key: YARN-8000 > URL: https://issues.apache.org/jira/browse/YARN-8000 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Fix For: 3.1.0 > > Attachments: Screen Shot 2018-03-08 at 1.37.07 PM.png, > YARN-8000.001.patch, YARN-8000.002.patch, YARN-8000.003.patch, > YARN-8000.004.patch > > > Yarn Service: component instance name shows up as component name in container > record -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5028) RMStateStore should trim down app state for completed applications
[
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-5028:
-
Fix Version/s: (was: 3.2.0)
> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch,
> YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch,
> YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch,
> YARN-5028.007-addendum.patch, YARN-5028.007.patch
>
>
> RMStateStore stores enough information to recover applications in case of a
> restart. The store also retains this information for completed applications
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all
> the information we store today to serve application status; for instance, we
> don't need the {{ApplicationSubmissionContext}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7947) Capacity Scheduler intra-queue preemption can NPE for non-schedulable apps
[ https://issues.apache.org/jira/browse/YARN-7947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7947: - Fix Version/s: (was: 3.2.0) > Capacity Scheduler intra-queue preemption can NPE for non-schedulable apps > -- > > Key: YARN-7947 > URL: https://issues.apache.org/jira/browse/YARN-7947 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, scheduler preemption >Affects Versions: 2.9.0, 2.8.3, 3.0.0, 3.1.0 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Fix For: 3.1.0, 2.10.0, 2.9.1, 2.8.4, 3.0.2 > > Attachments: YARN-7947.001.patch > > > Intra-queue preemption policy can cause NPE for pending users with no > schedulable apps. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5015) Support sliding window retry capability for container restart
[ https://issues.apache.org/jira/browse/YARN-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5015: - Fix Version/s: (was: 3.2.0) > Support sliding window retry capability for container restart > -- > > Key: YARN-5015 > URL: https://issues.apache.org/jira/browse/YARN-5015 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Chandni Singh >Priority: Major > Labels: oct16-medium > Fix For: 3.1.0 > > Attachments: YARN-5015.01.patch, YARN-5015.02.patch, > YARN-5015.03.patch, YARN-5015.04.patch, YARN-5015.05.patch, > YARN-5015.06.patch, YARN-5015.07.patch, YARN-5015.08.patch > > > We support sliding window retry policy for AM restarts (Introduced in > YARN-611). Similar sliding window retry policy is needed for container > restarts. > With this change, we can introduce a common class for > SlidingWindowRetryPolicy ( suggested by [~vvasudev] in the comments) and > integrate it to container restart. > In a subsequent jira, we can modify the AM code to use > SlidingWindowRetryPolicy which will unify the AM and container restart code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7919) Refactor timelineservice-hbase module into submodules
[ https://issues.apache.org/jira/browse/YARN-7919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7919: - Fix Version/s: (was: 3.2.0) > Refactor timelineservice-hbase module into submodules > - > > Key: YARN-7919 > URL: https://issues.apache.org/jira/browse/YARN-7919 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineservice >Affects Versions: 3.0.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Fix For: 3.1.0, 2.10.0 > > Attachments: YARN-7919-branch-2.05.patch, YARN-7919.00.patch, > YARN-7919.01.patch, YARN-7919.02.patch, YARN-7919.03.patch, > YARN-7919.04.patch, YARN-7919.05.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7972) Support inter-app placement constraints for allocation tags by application ID
[ https://issues.apache.org/jira/browse/YARN-7972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7972: - Fix Version/s: (was: 3.2.0) > Support inter-app placement constraints for allocation tags by application ID > - > > Key: YARN-7972 > URL: https://issues.apache.org/jira/browse/YARN-7972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-7972.001.patch, YARN-7972.002.patch, > YARN-7972.003.patch, YARN-7972.004.patch, YARN-7972.005.patch, > YARN-7972.006.patch, YARN-7972.007.patch > > > Per discussion in [this > comment|https://issues.apache.org/jira/browse/YARN-6599focusedCommentId=16319662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16319662] > in YARN-6599, we need to support inter-app PC for allocation tags. > This will help to do better placement when dealing with potential competing > resource applications, e.g don't place two tensorflow workers from two > different applications on one same node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7937) Fix http method name in Cluster Application Timeout Update API example request
[ https://issues.apache.org/jira/browse/YARN-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7937: - Fix Version/s: (was: 3.2.0) > Fix http method name in Cluster Application Timeout Update API example request > -- > > Key: YARN-7937 > URL: https://issues.apache.org/jira/browse/YARN-7937 > Project: Hadoop YARN > Issue Type: Bug > Components: docs, documentation >Affects Versions: 2.9.0, 3.0.0 >Reporter: Charan Hebri >Assignee: Charan Hebri >Priority: Minor > Fix For: 3.1.0, 2.10.0, 3.0.1 > > Attachments: YARN-7937.001.patch > > > In section, Cluster Application Timeout Update API, > https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Application_Timeout_Update_API > the example requests for both XML and JSON formats show "GET" as the http > method. This should actually be a PUT. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7955) Calling stop on an already stopped service says "Successfully stopped service"
[
https://issues.apache.org/jira/browse/YARN-7955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7955:
-
Fix Version/s: (was: 3.2.0)
> Calling stop on an already stopped service says "Successfully stopped service"
> --
>
> Key: YARN-7955
> URL: https://issues.apache.org/jira/browse/YARN-7955
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7955.001.patch, YARN-7955.002.patch
>
>
> If you invoke "yarn app -stop " on an already stopped service
> it confusingly responds with message "Successfully stopped service
> ". It should say "Service is already stopped".
> The same is seen with the REST API PUT request with data \{ "state":
> "STOPPED"}, the response is 200 OK and diagnostics with same message
> "Successfully stopped service ". It should return 400 Bad
> Request with message "Service is already stopped".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7657) Queue Mapping could provide options to provide 'user' specific auto-created queues under a specified group parent queue
[ https://issues.apache.org/jira/browse/YARN-7657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7657: - Fix Version/s: (was: 3.2.0) > Queue Mapping could provide options to provide 'user' specific auto-created > queues under a specified group parent queue > --- > > Key: YARN-7657 > URL: https://issues.apache.org/jira/browse/YARN-7657 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-7657.1.patch, YARN-7657.2.patch, YARN-7657.3.patch, > YARN-7657.4.patch > > > Current Queue-Mapping only provides %user as an option for 'user' specific > queues as u:%user:%user. We can also support %user with group as > 'g:makerting-group:marketing.%user' and user specific queues can be > automatically created under a group queue in this case. > cc [~leftnoteasy] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7915) Trusted image log message repeated multiple times
[
https://issues.apache.org/jira/browse/YARN-7915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7915:
-
Fix Version/s: (was: 3.2.0)
> Trusted image log message repeated multiple times
> --
>
> Key: YARN-7915
> URL: https://issues.apache.org/jira/browse/YARN-7915
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: Eric Badger
>Assignee: Shane Kumpf
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7915.001.patch
>
>
> Everytime we call {{check_trusted_image()}} we get a log message saying
> whether the image is trusted or not. In the case where it is trusted, the log
> message will get printed once for every call to the function. It's
> unnecessarily repetitive. I'm not really sure we need the log at all if the
> image is trusted. Maybe only log if it isn't trusted
> {noformat}
> Application application_1518201929288_0010 failed 3 times due to AM Container
> for appattempt_1518201929288_0010_03 exited with exitCode: 1
> Failing this attempt.Diagnostics: [2018-02-09 20:32:09.391]Exception from
> container-launch.
> Container id: container_1518201929288_0010_03_01
> Exit code: 1
> Exception message: image: foo/bar is trusted in foo registry.
> image: foo/bar is trusted in foo registry.
> image: foo/bar is trusted in foo registry.
> Docker container exit code was not zero: 1
> Unable to read from docker logs(ferror, feof): 0 1
> Shell output: main : command provided 4
> main : run as user is ebadger
> main : requested yarn user is ebadger
> Creating script paths...
> Creating local dirs...
> Getting exit code file...
> Changing effective user to root...
> Launching docker container...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7636) Re-reservation count may overflow when cluster resource exhausted for a long time
[
https://issues.apache.org/jira/browse/YARN-7636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7636:
-
Fix Version/s: (was: 3.2.0)
> Re-reservation count may overflow when cluster resource exhausted for a long
> time
> --
>
> Key: YARN-7636
> URL: https://issues.apache.org/jira/browse/YARN-7636
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.1.0, 2.9.1
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Fix For: 3.1.0, 2.9.1, 3.0.2
>
> Attachments: YARN-7636.001.patch, YARN-7636.002.patch,
> YARN-7636.003.patch
>
>
> This happens on our production cluster twice, when a request cannot be
> satisfied for a long time, it continually triggers the re-reservation and
> eventually caused the overflow. This will crash the scheduler.
> Exception stack:
> {noformat}
> java.lang.IllegalArgumentException: Overflow adding 1 occurrences to a count
> of 2147483647
> at
> com.google.common.collect.ConcurrentHashMultiset.add(ConcurrentHashMultiset.java:246)
> at
> com.google.common.collect.AbstractMultiset.add(AbstractMultiset.java:80)
> at
> com.google.common.collect.ConcurrentHashMultiset.add(ConcurrentHashMultiset.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.addReReservation(SchedulerApplicationAttempt.java:406)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.reserve(SchedulerApplicationAttempt.java:555)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.reserve(FiCaSchedulerApp.java:1076)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.apply(FiCaSchedulerApp.java:795)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.tryCommit(CapacityScheduler.java:2770)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$ResourceCommitterService.run(CapacityScheduler.java:546)
> {noformat}
> Refer to handling of SchedulerApplicationAttempt#addSchedulingOpportunity, we
> can ignore this exception to avoid this problem.
> This problem may happens in
> SchedulerApplicationAttempt#addMissedNonPartitionedRequestSchedulingOpportunity,
> fix it in the same way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8002) Support NOT_SELF and ALL namespace types for allocation tag
[ https://issues.apache.org/jira/browse/YARN-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405230#comment-16405230 ] Hudson commented on YARN-8002: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13855 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13855/]) YARN-8002. Support NOT_SELF and ALL namespace types for allocation tag. (wangda: rev a08921ca6cb1dad98935808c8f474b654f861263) * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Evaluable.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/TargetApplications.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/Evaluable.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/PlacementConstraintsUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/AllocationTagsManager.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/AllocationTagNamespace.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/AllocationTagNamespace.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/TestSingleConstraintAppPlacementAllocator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/algorithm/TestLocalAllocationTagsManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/resource/PlacementConstraints.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/TestAllocationTagsManager.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/AllocationTags.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/TestPlacementConstraintsUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SingleConstraintAppPlacementAllocator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/AllocationTags.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/AllocationTagNamespaceType.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestSchedulingRequestContainerAllocation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/TestAllocationTagsNamespace.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/TargetApplications.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/InvalidAllocationTagException.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/algorithm/LocalAllocationTagsManager.java > Support NOT_SELF and ALL namespace types for allocation tag > --- > > Key: YARN-8002 > URL: https://issues.apache.org/jira/browse/YARN-8002 > Project: Hadoop YARN > Issue
[jira] [Commented] (YARN-7581) HBase filters are not constructed correctly in ATSv2
[ https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405226#comment-16405226 ] Vrushali C commented on YARN-7581: -- Also, Rohith was +1 on this patch, so perhaps I can commit it later today. Or [~rohithsharma], please feel free to commit it in case I don't get to it by today. > HBase filters are not constructed correctly in ATSv2 > > > Key: YARN-7581 > URL: https://issues.apache.org/jira/browse/YARN-7581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-7581-YARN-7055.04.patch, YARN-7581.00.patch, > YARN-7581.01.patch, YARN-7581.02.patch, YARN-7581.03.patch, > YARN-7581.04.patch, YARN-7581.05.patch > > > Post YARN-7346, > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters() and > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters() > start to fail when hbase.profile is set to 2.0) > *Error Message* > [ERROR] Failures: > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1266 > expected:<2> but was:<0> > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1523 > expected:<1> but was:<0> -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8024) LOG in class MaxRunningAppsEnforcer is initialized with a faulty class FairScheduler
[ https://issues.apache.org/jira/browse/YARN-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8024: - Fix Version/s: 3.1.0 > LOG in class MaxRunningAppsEnforcer is initialized with a faulty class > FairScheduler > - > > Key: YARN-8024 > URL: https://issues.apache.org/jira/browse/YARN-8024 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Sen Zhao >Priority: Major > Labels: newbie++ > Fix For: 3.1.0, 3.2.0 > > Attachments: YARN-8024.001.patch > > > It should be initialized with class MaxRunningAppsEnforcer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5028) RMStateStore should trim down app state for completed applications
[
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-5028:
-
Fix Version/s: 3.1.0
> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Fix For: 3.1.0, 3.2.0
>
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch,
> YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch,
> YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch,
> YARN-5028.007-addendum.patch, YARN-5028.007.patch
>
>
> RMStateStore stores enough information to recover applications in case of a
> restart. The store also retains this information for completed applications
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all
> the information we store today to serve application status; for instance, we
> don't need the {{ApplicationSubmissionContext}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7657) Queue Mapping could provide options to provide 'user' specific auto-created queues under a specified group parent queue
[ https://issues.apache.org/jira/browse/YARN-7657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7657: - Fix Version/s: 3.1.0 > Queue Mapping could provide options to provide 'user' specific auto-created > queues under a specified group parent queue > --- > > Key: YARN-7657 > URL: https://issues.apache.org/jira/browse/YARN-7657 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: YARN-7657.1.patch, YARN-7657.2.patch, YARN-7657.3.patch, > YARN-7657.4.patch > > > Current Queue-Mapping only provides %user as an option for 'user' specific > queues as u:%user:%user. We can also support %user with group as > 'g:makerting-group:marketing.%user' and user specific queues can be > automatically created under a group queue in this case. > cc [~leftnoteasy] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7919) Refactor timelineservice-hbase module into submodules
[ https://issues.apache.org/jira/browse/YARN-7919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7919: - Fix Version/s: 3.1.0 > Refactor timelineservice-hbase module into submodules > - > > Key: YARN-7919 > URL: https://issues.apache.org/jira/browse/YARN-7919 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineservice >Affects Versions: 3.0.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Fix For: 3.1.0, 2.10.0, 3.2.0 > > Attachments: YARN-7919-branch-2.05.patch, YARN-7919.00.patch, > YARN-7919.01.patch, YARN-7919.02.patch, YARN-7919.03.patch, > YARN-7919.04.patch, YARN-7919.05.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7446) Docker container privileged mode and --user flag contradict each other
[ https://issues.apache.org/jira/browse/YARN-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7446: - Fix Version/s: 3.1.0 > Docker container privileged mode and --user flag contradict each other > -- > > Key: YARN-7446 > URL: https://issues.apache.org/jira/browse/YARN-7446 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0, 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: YARN-7446.001.patch, YARN-7446.002.patch, > YARN-7446.003.patch, YARN-7446.004.patch > > > In the current implementation, when privileged=true, --user flag is also > passed to docker for launching container. In reality, the container has no > way to use root privileges unless there is sticky bit or sudoers in the image > for the specified user to gain privileges again. To avoid duplication of > dropping and reacquire root privileges, we can reduce the duplication of > specifying both flag. When privileged mode is enabled, --user flag should be > omitted. When non-privileged mode is enabled, --user flag is supplied. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7955) Calling stop on an already stopped service says "Successfully stopped service"
[
https://issues.apache.org/jira/browse/YARN-7955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7955:
-
Fix Version/s: 3.1.0
> Calling stop on an already stopped service says "Successfully stopped service"
> --
>
> Key: YARN-7955
> URL: https://issues.apache.org/jira/browse/YARN-7955
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
> Fix For: 3.1.0, 3.2.0
>
> Attachments: YARN-7955.001.patch, YARN-7955.002.patch
>
>
> If you invoke "yarn app -stop " on an already stopped service
> it confusingly responds with message "Successfully stopped service
> ". It should say "Service is already stopped".
> The same is seen with the REST API PUT request with data \{ "state":
> "STOPPED"}, the response is 200 OK and diagnostics with same message
> "Successfully stopped service ". It should return 400 Bad
> Request with message "Service is already stopped".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7835) [Atsv2] Race condition in NM while publishing events if second attempt is launched on the same node
[ https://issues.apache.org/jira/browse/YARN-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7835: - Fix Version/s: 3.1.0 > [Atsv2] Race condition in NM while publishing events if second attempt is > launched on the same node > --- > > Key: YARN-7835 > URL: https://issues.apache.org/jira/browse/YARN-7835 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Critical > Fix For: 3.1.0, 2.10.0, 3.2.0 > > Attachments: YARN-7835.001.patch, YARN-7835.002.patch, > YARN-7835.003.patch, YARN-7835.004.patch > > > It is observed race condition that if master container is killed for some > reason and launched on same node then NMTimelinePublisher doesn't add > timelineClient. But once completed container for 1st attempt has come then > NMTimelinePublisher removes the timelineClient. > It causes all subsequent event publishing from different client fails to > publish with exception Application is not found. ! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7581) HBase filters are not constructed correctly in ATSv2
[ https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405225#comment-16405225 ] Vrushali C commented on YARN-7581: -- Yes, thanks, +1 on patch 05. The adding of info CF is pre-existing, so let's keep that. We may want to revisit it in case we see slow performance but that's not related to this patch. > HBase filters are not constructed correctly in ATSv2 > > > Key: YARN-7581 > URL: https://issues.apache.org/jira/browse/YARN-7581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-7581-YARN-7055.04.patch, YARN-7581.00.patch, > YARN-7581.01.patch, YARN-7581.02.patch, YARN-7581.03.patch, > YARN-7581.04.patch, YARN-7581.05.patch > > > Post YARN-7346, > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters() and > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters() > start to fail when hbase.profile is set to 2.0) > *Error Message* > [ERROR] Failures: > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1266 > expected:<2> but was:<0> > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1523 > expected:<1> but was:<0> -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7891) LogAggregationIndexedFileController should support read from HAR file
[ https://issues.apache.org/jira/browse/YARN-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7891: - Fix Version/s: 3.1.0 > LogAggregationIndexedFileController should support read from HAR file > - > > Key: YARN-7891 > URL: https://issues.apache.org/jira/browse/YARN-7891 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: YARN-7891.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7934) [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos
[ https://issues.apache.org/jira/browse/YARN-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7934: - Fix Version/s: 3.1.0 > [GQ] Refactor preemption calculators to allow overriding for Federation > Global Algos > > > Key: YARN-7934 > URL: https://issues.apache.org/jira/browse/YARN-7934 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: YARN-7934.v1.patch, YARN-7934.v2.patch, > YARN-7934.v3.patch, YARN-7934.v4.patch > > > This Jira tracks minimal changes in the capacity scheduler preemption > mechanics that allow for sub-classing and overriding of certain behaviors, > which we use to implement federation global algorithms, e.g., in YARN-7403. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7985) Service name is validated twice in ServiceClient when a service is created
[
https://issues.apache.org/jira/browse/YARN-7985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7985:
-
Fix Version/s: 3.1.0
> Service name is validated twice in ServiceClient when a service is created
> --
>
> Key: YARN-7985
> URL: https://issues.apache.org/jira/browse/YARN-7985
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Affects Versions: 3.1.0
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Trivial
> Fix For: 3.1.0, 3.2.0
>
> Attachments: YARN-7985.001.patch
>
>
> In ServiceClient.actionCreate() :
> {code:java}
> ServiceApiUtil.validateNameFormat(serviceName, getConfig());
> ServiceApiUtil.validateAndResolveService(service, fs, getConfig());{code}
> However, the ServiceApiUtil.validateAndResolveService(...), also validates
> the service name
> {code:java}
> if (StringUtils.isEmpty(service.getName())) {
> throw new IllegalArgumentException(
> RestApiErrorMessages.ERROR_APPLICATION_NAME_INVALID);
> }
> validateNameFormat(service.getName(), conf);{code}
> Also, {{ServiceClientTest}} which is a mock client for the ApiServer, can
> perform the actual validation which is performed by the {{ServiceClient}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5015) Support sliding window retry capability for container restart
[ https://issues.apache.org/jira/browse/YARN-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5015: - Fix Version/s: 3.1.0 > Support sliding window retry capability for container restart > -- > > Key: YARN-5015 > URL: https://issues.apache.org/jira/browse/YARN-5015 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Chandni Singh >Priority: Major > Labels: oct16-medium > Fix For: 3.1.0, 3.2.0 > > Attachments: YARN-5015.01.patch, YARN-5015.02.patch, > YARN-5015.03.patch, YARN-5015.04.patch, YARN-5015.05.patch, > YARN-5015.06.patch, YARN-5015.07.patch, YARN-5015.08.patch > > > We support sliding window retry policy for AM restarts (Introduced in > YARN-611). Similar sliding window retry policy is needed for container > restarts. > With this change, we can introduce a common class for > SlidingWindowRetryPolicy ( suggested by [~vvasudev] in the comments) and > integrate it to container restart. > In a subsequent jira, we can modify the AM code to use > SlidingWindowRetryPolicy which will unify the AM and container restart code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7954) Component status stays "Ready" when yarn service is stopped
[
https://issues.apache.org/jira/browse/YARN-7954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7954:
-
Fix Version/s: 3.1.0
> Component status stays "Ready" when yarn service is stopped
> ---
>
> Key: YARN-7954
> URL: https://issues.apache.org/jira/browse/YARN-7954
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Gour Saha
>Priority: Major
> Fix For: 3.1.0, 3.2.0
>
> Attachments: YARN-7954.001.patch, YARN-7954.002.patch
>
>
> Steps:
> 1) Launch yarn service application
> 2) Stop application
> 3) Run get status from yarn cli
> {code}
> [hdpuser@cn005 sleeper]$ yarn app -status yesha-sleeper
> WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of
> YARN_LOG_DIR.
> WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of
> YARN_LOGFILE.
> WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of
> YARN_PID_DIR.
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 18/02/16 10:54:37 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/02/16 10:54:37 INFO client.RMProxy: Connecting to ResourceManager at
> xxx/xx.xx.xx.xx:8050
> 18/02/16 10:54:37 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xx.xx.xx.xx:10200
> 18/02/16 10:54:37 INFO client.RMProxy: Connecting to ResourceManager at
> xxx/xx.xx.xx.xx:8050
> 18/02/16 10:54:37 INFO client.AHSProxy: Connecting to Application History
> server at xxx/xx.xx.xx.xx:10200
> 18/02/16 10:54:38 INFO util.log: Logging initialized @1957ms
> {"name":"yesha-sleeper","lifetime":-1,"components":[],"configuration":{"properties":{},"env":{},"files":[]},"state":"STOPPED","quicklinks":{},"kerberos_principal":{}}
> {code}
> 4) Validate UI2 for service status
> Here, Yarn service status is marked as "finished". However, components status
> still shows Ready.
> On stopping yarn service, component status should be updated to "Stop"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7952) RM should be able to recover log aggregation status after restart/fail-over
[ https://issues.apache.org/jira/browse/YARN-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7952: - Fix Version/s: 3.1.0 > RM should be able to recover log aggregation status after restart/fail-over > --- > > Key: YARN-7952 > URL: https://issues.apache.org/jira/browse/YARN-7952 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: YARN-7952-poc.patch, YARN-7952.1.patch, > YARN-7952.2.patch, YARN-7952.3.patch, YARN-7952.3.patch, YARN-7952.5.patch, > YARN-7952.6.patch, YARN-7952.7.patch, YARN-7952.8.patch, YARN-7952.9.patch > > > Right now, the NM would send its own log aggregation status to RM > periodically to RM. And RM would aggregate the status for each application, > but it will not generate the final status until a client call(from web ui or > cli) trigger it. But RM never persists the log aggregation status. So, when > RM restarts/fails over, the log aggregation status will become “NOT_STARTED”. > This is confusing, maybe we should change it to “NOT_AVAILABLE” (will create > a separate ticket for this). Anyway, we need to persist the log aggregation > status for the future use. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7346) Add a profile to allow optional compilation for ATSv2 with HBase-2.0
[ https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7346: - Fix Version/s: 3.1.0 > Add a profile to allow optional compilation for ATSv2 with HBase-2.0 > > > Key: YARN-7346 > URL: https://issues.apache.org/jira/browse/YARN-7346 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Haibo Chen >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: YARN-7346.00.patch, YARN-7346.01.patch, > YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, > YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, > YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08-incremental.patch, > YARN-7346.08.patch, YARN-7346.09.patch, YARN-7346.10.patch, > YARN-7346.11.patch, YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, > YARN-7581.prelim.patch, > hadoop-yarn-server-timelineservice-hbase-server-1-findbugsXml.xml, > hadoop-yarn-server-timelineservice-hbase-server-1-javadoc-report.txt, > hadoop-yarn-server-timelineservice-hbase-server-2-findbugsXml.xml, > hadoop-yarn-server-timelineservice-hbase-server-2-javadoc-report.txt > > > When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, > I got the following errors: > [https://pastebin.com/Ms4jYEVB] > This issue is to fix the compilation errors. > The scope of the Jira is to add a profile to allow optional compilation for > ATSv2 with HBase2.0. The default compilation for trunk will still be for > hbase 1.2.6. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8040) [UI2] New YARN UI webapp does not respect current pathname for REST api
[ https://issues.apache.org/jira/browse/YARN-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8040: - Fix Version/s: 3.2.0 > [UI2] New YARN UI webapp does not respect current pathname for REST api > --- > > Key: YARN-8040 > URL: https://issues.apache.org/jira/browse/YARN-8040 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8040.001.patch > > > When ui2 is accessed behind proxy like knox/nginx, trailing path name should > not be skipped. However trim of "ui2" if its there. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8011) TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart fails sometimes in trunk
[
https://issues.apache.org/jira/browse/YARN-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-8011:
-
Fix Version/s: 3.1.0
> TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart
> fails sometimes in trunk
> ---
>
> Key: YARN-8011
> URL: https://issues.apache.org/jira/browse/YARN-8011
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Minor
> Fix For: 3.1.0, 3.2.0
>
> Attachments: YARN-8011.001.patch, YARN-8011.002.patch
>
>
> TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart
> often pass, but the following errors sometimes occur:
> {noformat}
> java.lang.AssertionError:
> Expected :15360
> Actual :14336
>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService.verifyMetrics(TestOpportunisticContainerAllocatorAMService.java:732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService.testContainerPromoteAndDemoteBeforeContainerStart(TestOpportunisticContainerAllocatorAMService.java:330)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
>
> This problem is caused by that deducting resource is a little behind the
> assertion. To solve this problem, It can sleep a while before this assertion
> as below.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7929) Support to set container execution type in SLS
[ https://issues.apache.org/jira/browse/YARN-7929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7929: - Fix Version/s: 3.1.0 > Support to set container execution type in SLS > -- > > Key: YARN-7929 > URL: https://issues.apache.org/jira/browse/YARN-7929 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: YARN-7929.001.patch, YARN-7929.0010.patch, > YARN-7929.002.patch, YARN-7929.003.patch, YARN-7929.004.patch, > YARN-7929.005.patch, YARN-7929.006.patch, YARN-7929.007.patch, > YARN-7929.008.patch, YARN-7929.009.patch > > > SLS currently support three tracetype, SYNTH, SLS and RUMEN, but trace file > can not set execution type of container. > This jira will introduce execution type in SLS to help better simulation. > This will help the perf testing with regarding to the Opportunistic > Containers. > RUMEN has default execution type GUARANTEED > SYNTH set execution type by field map_execution_type and > reduce_execution_type > SLS set execution type by field container.execution_type > For compatibility set GUARANTEED as default value when not setting above > fields in trace file -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7626) Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount
[ https://issues.apache.org/jira/browse/YARN-7626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7626: - Fix Version/s: 3.1.0 > Allow regular expression matching in container-executor.cfg for devices and > named docker volumes mount > -- > > Key: YARN-7626 > URL: https://issues.apache.org/jira/browse/YARN-7626 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Zian Chen >Assignee: Zian Chen >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: YARN-7626.001.patch, YARN-7626.002.patch, > YARN-7626.003.patch, YARN-7626.004.patch, YARN-7626.005.patch, > YARN-7626.006.patch, YARN-7626.007.patch, YARN-7626.008.patch, > YARN-7626.009.patch, YARN-7626.010.patch, YARN-7626.011.patch > > > Currently when we config some of the GPU devices related fields (like ) in > container-executor.cfg, these fields are generated based on different driver > versions or GPU device names. We want to enable regular expression matching > so that user don't need to manually set up these fields when config > container-executor.cfg, -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7915) Trusted image log message repeated multiple times
[
https://issues.apache.org/jira/browse/YARN-7915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7915:
-
Fix Version/s: 3.1.0
> Trusted image log message repeated multiple times
> --
>
> Key: YARN-7915
> URL: https://issues.apache.org/jira/browse/YARN-7915
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: Eric Badger
>Assignee: Shane Kumpf
>Priority: Major
> Fix For: 3.1.0, 3.2.0
>
> Attachments: YARN-7915.001.patch
>
>
> Everytime we call {{check_trusted_image()}} we get a log message saying
> whether the image is trusted or not. In the case where it is trusted, the log
> message will get printed once for every call to the function. It's
> unnecessarily repetitive. I'm not really sure we need the log at all if the
> image is trusted. Maybe only log if it isn't trusted
> {noformat}
> Application application_1518201929288_0010 failed 3 times due to AM Container
> for appattempt_1518201929288_0010_03 exited with exitCode: 1
> Failing this attempt.Diagnostics: [2018-02-09 20:32:09.391]Exception from
> container-launch.
> Container id: container_1518201929288_0010_03_01
> Exit code: 1
> Exception message: image: foo/bar is trusted in foo registry.
> image: foo/bar is trusted in foo registry.
> image: foo/bar is trusted in foo registry.
> Docker container exit code was not zero: 1
> Unable to read from docker logs(ferror, feof): 0 1
> Shell output: main : command provided 4
> main : run as user is ebadger
> main : requested yarn user is ebadger
> Creating script paths...
> Creating local dirs...
> Getting exit code file...
> Changing effective user to root...
> Launching docker container...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7972) Support inter-app placement constraints for allocation tags by application ID
[ https://issues.apache.org/jira/browse/YARN-7972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7972: - Fix Version/s: 3.1.0 > Support inter-app placement constraints for allocation tags by application ID > - > > Key: YARN-7972 > URL: https://issues.apache.org/jira/browse/YARN-7972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: YARN-7972.001.patch, YARN-7972.002.patch, > YARN-7972.003.patch, YARN-7972.004.patch, YARN-7972.005.patch, > YARN-7972.006.patch, YARN-7972.007.patch > > > Per discussion in [this > comment|https://issues.apache.org/jira/browse/YARN-6599focusedCommentId=16319662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16319662] > in YARN-6599, we need to support inter-app PC for allocation tags. > This will help to do better placement when dealing with potential competing > resource applications, e.g don't place two tensorflow workers from two > different applications on one same node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7995) Remove unnecessary boxings and unboxings from PlacementConstraintParser.java
[
https://issues.apache.org/jira/browse/YARN-7995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7995:
-
Fix Version/s: 3.1.0
> Remove unnecessary boxings and unboxings from PlacementConstraintParser.java
>
>
> Key: YARN-7995
> URL: https://issues.apache.org/jira/browse/YARN-7995
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Sen Zhao
>Priority: Minor
> Labels: newbie
> Fix For: 3.1.0, 3.2.0
>
> Attachments: YARN-7995.001.patch
>
>
> {code}
> String maxCardinalityStr = resetElements.pop();
> Integer max = toInt(maxCardinalityStr);
> String minCardinalityStr = resetElements.pop();
> Integer min = toInt(minCardinalityStr);
> {code}
> {{toInt(String)}} does not return null, so {{Integer}} should be replaced
> with {{int}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8000) Yarn Service: component instance name shows up as component name in container record
[ https://issues.apache.org/jira/browse/YARN-8000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8000: - Fix Version/s: 3.1.0 > Yarn Service: component instance name shows up as component name in container > record > - > > Key: YARN-8000 > URL: https://issues.apache.org/jira/browse/YARN-8000 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: Screen Shot 2018-03-08 at 1.37.07 PM.png, > YARN-8000.001.patch, YARN-8000.002.patch, YARN-8000.003.patch, > YARN-8000.004.patch > > > Yarn Service: component instance name shows up as component name in container > record -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6936) [Atsv2] Retrospect storing entities into sub application table from client perspective
[ https://issues.apache.org/jira/browse/YARN-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6936: - Issue Type: Sub-task (was: Bug) Parent: YARN-7055 > [Atsv2] Retrospect storing entities into sub application table from client > perspective > -- > > Key: YARN-6936 > URL: https://issues.apache.org/jira/browse/YARN-6936 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-6936.000.patch > > > Currently YARN-6734 stores entities into sub application table only if doAs > user and submitted users are different. This holds good for Tez kind of use > cases. But AM runs as same as submitted user like MR also need to store > entities in sub application table so that it could read entities without > application id. > This would be a point of concern later stages when ATSv2 is deployed into > production. This JIRA is to retrospect decision of storing entities into sub > application table based on client side configuration driven rather than user > driven. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7581) HBase filters are not constructed correctly in ATSv2
[ https://issues.apache.org/jira/browse/YARN-7581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405218#comment-16405218 ] Haibo Chen commented on YARN-7581: -- [~vrushalic] Are you okay with the latest patch? > HBase filters are not constructed correctly in ATSv2 > > > Key: YARN-7581 > URL: https://issues.apache.org/jira/browse/YARN-7581 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-7581-YARN-7055.04.patch, YARN-7581.00.patch, > YARN-7581.01.patch, YARN-7581.02.patch, YARN-7581.03.patch, > YARN-7581.04.patch, YARN-7581.05.patch > > > Post YARN-7346, > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters() and > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters() > start to fail when hbase.profile is set to 2.0) > *Error Message* > [ERROR] Failures: > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1266 > expected:<2> but was:<0> > [ERROR] > TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1523 > expected:<1> but was:<0> -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8020) when DRF is used, preemption does not trigger due to incorrect idealAssigned
[
https://issues.apache.org/jira/browse/YARN-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405166#comment-16405166
]
Eric Payne commented on YARN-8020:
--
[~leftnoteasy], sorry for the delay.
{quote}explain why preemption doesn't happen for the case you mentioned
{quote}
As it turns out, the corner case I'm running into is not related to DRF. It has
the same behavior with the default resource calculator.
The use case is this:
- QueueA is preemptable and is running App1 which is consuming the entire
cluster
- App2 is submitted to QueueB with container requests where each container is
larger then the user limit for QueueB.
In this case, preemption will not occur.
DETAILS:
- Cluster size: 20G
- Cluster Min container size: 1G
- QueueA capacity: 10G
- QueueB capacity: 10G
- QueueB MULP: 10%
ACTIONS:
- App1 running in QueueA consumes 20G
- App2 is submitted to QueueB with AM size 1G and map container sizes 4G.
- App2's Max Resource is 1G when it is requesting the AM container (10% of 10G
== 1G). The preemption monitor sees that the pending request is 1G and that
App2's headroom is 1G, so it preempts 1G from App1 in QueueA.
- The Capacity Scheduler assigns 1G to App2 in QueueB. App2 begins running the
AM container.
- App2 requests several map containers at 4G each. App2's Max Resource is
computed to be 2G. (((active user's used resources/# active users) + min
container size) == (1G/1 + 1G) == 2G)). This leaves 1G of headroom for App2.
- The preemption monitor sees that the requested container size for App2 is 4G
which is larger than the 1G headroom, so the preemption monitor does not
preempt.
Technically, this behavior is slightly out of sync with the way the capacity
scheduler assigns containers. As long as the headroom for an app is 0 or more,
the capacity scheduler will assign one more container, no matter how big the
container is, so the preemption monitor should go ahead and preempt in this
case. I'm not sure I want it to, though, because it's better to be conservative
than to preempt when it should not.
[~kyungwan nam], on what version of YARN are you seeing this problem? I am not
seeing any DRF-related issues in 2.8 or 3.x.
> when DRF is used, preemption does not trigger due to incorrect idealAssigned
>
>
> Key: YARN-8020
> URL: https://issues.apache.org/jira/browse/YARN-8020
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: kyungwan nam
>Priority: Major
>
> I’ve met that Inter Queue Preemption does not work.
> It happens when DRF is used and submitting application with a large number of
> vcores.
> IMHO, idealAssigned can be set incorrectly by following code.
> {code}
> // This function "accepts" all the resources it can (pending) and return
> // the unused ones
> Resource offer(Resource avail, ResourceCalculator rc,
> Resource clusterResource, boolean considersReservedResource) {
> Resource absMaxCapIdealAssignedDelta = Resources.componentwiseMax(
> Resources.subtract(getMax(), idealAssigned),
> Resource.newInstance(0, 0));
> // accepted = min{avail,
> // max - assigned,
> // current + pending - assigned,
> // # Make sure a queue will not get more than max of its
> // # used/guaranteed, this is to make sure preemption won't
> // # happen if all active queues are beyond their guaranteed
> // # This is for leaf queue only.
> // max(guaranteed, used) - assigned}
> // remain = avail - accepted
> Resource accepted = Resources.min(rc, clusterResource,
> absMaxCapIdealAssignedDelta,
> Resources.min(rc, clusterResource, avail, Resources
> /*
>* When we're using FifoPreemptionSelector (considerReservedResource
>* = false).
>*
>* We should deduct reserved resource from pending to avoid
> excessive
>* preemption:
>*
>* For example, if an under-utilized queue has used = reserved = 20.
>* Preemption policy will try to preempt 20 containers (which is not
>* satisfied) from different hosts.
>*
>* In FifoPreemptionSelector, there's no guarantee that preempted
>* resource can be used by pending request, so policy will preempt
>* resources repeatly.
>*/
> .subtract(Resources.add(getUsed(),
> (considersReservedResource ? pending : pendingDeductReserved)),
> idealAssigned)));
> {code}
> let’s say,
> * cluster resource :
> * idealAssigned(assigned):
> * avail:
> * current:
> * pending:
[jira] [Commented] (YARN-7494) Add muti node lookup support for better placement
[ https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405157#comment-16405157 ] Sunil G commented on YARN-7494: --- Updating patch by correct api names where each scheduler can register policies to monitor by common MultiNodeManager. [~cheersyang] [~leftnoteasy] pls check the latest patch. > Add muti node lookup support for better placement > - > > Key: YARN-7494 > URL: https://issues.apache.org/jira/browse/YARN-7494 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > Attachments: YARN-7494.001.patch, YARN-7494.002.patch, > YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.v0.patch, > YARN-7494.v1.patch, multi-node-designProposal.png > > > Instead of single node, for effectiveness we can consider a multi node lookup > based on partition to start with. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7494) Add muti node lookup support for better placement
[ https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7494: -- Attachment: YARN-7494.004.patch > Add muti node lookup support for better placement > - > > Key: YARN-7494 > URL: https://issues.apache.org/jira/browse/YARN-7494 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > Attachments: YARN-7494.001.patch, YARN-7494.002.patch, > YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.v0.patch, > YARN-7494.v1.patch, multi-node-designProposal.png > > > Instead of single node, for effectiveness we can consider a multi node lookup > based on partition to start with. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8016) Refine PlacementRule interface and add a app-name queue mapping rule as an example
[
https://issues.apache.org/jira/browse/YARN-8016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405094#comment-16405094
]
Zian Chen commented on YARN-8016:
-
Just went through the failed cases and warnings about the second patch,
1. hadoop-yarn-api-warnings.html:
org.apache.hadoop.yarn.api.records.Resource.getResources() may expose internal
representation by returning Resource.resources. {color:#FF}Have no idea why
the second patch affects this part.{color}
2. hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html: Redundant
nullcheck of rule, which is known to be non-null in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules():
{color:#FF}will fix this issue in next patch{color}
3. For the failed UT cases, nearly all the cases were failed due to throw
exception when we call UserGroupMappingPlacementRule#initialize and didn't get
any newMappings. Looks like we can not force throwing the exception in
UserGroupMappingPlacementRule#initialize to ensure user to add
UserGroupMappingPlacementRule into List placementRules data
structure if absent. What we can do here is add UserGroupMappingPlacementRule
into placementRuleStrs and do the switch case check, but if we do no have
corresponding placement rule mapping values setting for parmeter
yarn.scheduler.capacity.queue-mappings, that should be totally fine. I suggest
we remove the throw expcetion inside UserGroupMappingPlacementRule#initialize
and return null if newMappings.size == 0 as we did it previously.
4. asflicense The patch generated 7 ASF License warnings: {color:#FF}have
no idea how to fix this issue.{color}
Could you share your opinions on these issues, [~leftnoteasy] ? Thank you!
> Refine PlacementRule interface and add a app-name queue mapping rule as an
> example
> --
>
> Key: YARN-8016
> URL: https://issues.apache.org/jira/browse/YARN-8016
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Zian Chen
>Assignee: Zian Chen
>Priority: Major
> Attachments: YARN-8016.001.patch, YARN-8016.002.patch
>
>
> After YARN-3635/YARN-6689, PlacementRule becomes a common interface which can
> be used by scheduler and can be dynamically updated by scheduler according to
> configs. There're some other works.
> - There's no way to initialize PlacementRule.
> - No example of PlacementRule except the user-group mapping one.
> This JIRA is targeted to refine PlacementRule interfaces and add another
> PlacementRule example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8050) Yarn High Availability ZooKeeper based discovery rather than explicit rm1,rm2 configs
[ https://issues.apache.org/jira/browse/YARN-8050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated YARN-8050: -- Summary: Yarn High Availability ZooKeeper based discovery rather than explicit rm1,rm2 configs (was: Yarn HA ZooKeeper based discovery rather than explicit rm1,rm2 configs) > Yarn High Availability ZooKeeper based discovery rather than explicit rm1,rm2 > configs > - > > Key: YARN-8050 > URL: https://issues.apache.org/jira/browse/YARN-8050 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, RM, yarn >Affects Versions: 2.9.1 >Reporter: Hari Sekhon >Priority: Major > > Improvement Request for Yarn Resource Manager HA to use ZooKeeper based > dynamic discovery rather than explicitly setting the Resource Manager > addresses via rm1,rm2 in the configs. > One proprietary Hadoop vendor already uses ZK for RM HA discovery - it makes > sense that the open source core should do the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8050) Yarn HA ZooKeeper based discovery rather than explicit rm1,rm2 configs
Hari Sekhon created YARN-8050: - Summary: Yarn HA ZooKeeper based discovery rather than explicit rm1,rm2 configs Key: YARN-8050 URL: https://issues.apache.org/jira/browse/YARN-8050 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, RM, yarn Affects Versions: 2.9.1 Reporter: Hari Sekhon Improvement Request for Yarn Resource Manager HA to use ZooKeeper based dynamic discovery rather than explicitly setting the Resource Manager addresses via rm1,rm2 in the configs. One proprietary Hadoop vendor already uses ZK for RM HA discovery - it makes sense that the open source core should do the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8016) Refine PlacementRule interface and add a app-name queue mapping rule as an example
[
https://issues.apache.org/jira/browse/YARN-8016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405070#comment-16405070
]
Wangda Tan commented on YARN-8016:
--
Thanks [~Zian Chen] for the update, I'm fine with keep methods in
CapacitySchedulerConfiguration, for the second comment:
bq. The second question is the test case for chain of placement rules. I
understand your suggestion is to like set up some cases which will let the
switch case condition inside CS#updatePlacementRules so that the case failed
with previous placement rule can still handle by the second placement rule, for
example, AppNamePlacementRule. However, the logic here is kind of strange, if
one case can enter into the first switch case which is UserGroupPlcaementRule,
then it means the mapping related to the case is for UserGroupMappingRule, here
in the switch case, we just initialize the rule based on the mapping, not doing
the getPlacementForApp method call, so the condition like we set user1 for the
mapping but we actually use user2 to consume the mapping will not happen cause
this is another story for getPlacementForApp related calls, not
updatePlacementRules.
IIRC, {{updatePlacementRules}} is invoked when reinitialize queue
configurations. And {{getPlacementForApp}} is invoked when do per-app queue
mapping. I'm not sure about what does your previous comment mean, could you
share any example?
Apart from UT failure, could u also take care of ASF license warning / findbugs
warning.
> Refine PlacementRule interface and add a app-name queue mapping rule as an
> example
> --
>
> Key: YARN-8016
> URL: https://issues.apache.org/jira/browse/YARN-8016
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Zian Chen
>Assignee: Zian Chen
>Priority: Major
> Attachments: YARN-8016.001.patch, YARN-8016.002.patch
>
>
> After YARN-3635/YARN-6689, PlacementRule becomes a common interface which can
> be used by scheduler and can be dynamically updated by scheduler according to
> configs. There're some other works.
> - There's no way to initialize PlacementRule.
> - No example of PlacementRule except the user-group mapping one.
> This JIRA is targeted to refine PlacementRule interfaces and add another
> PlacementRule example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8016) Refine PlacementRule interface and add a app-name queue mapping rule as an example
[ https://issues.apache.org/jira/browse/YARN-8016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405061#comment-16405061 ] Zian Chen commented on YARN-8016: - Hi [~leftnoteasy], I see several failed test cases here which related to my latest patch, let me address these cases first then you could help me review the patch. Thanks > Refine PlacementRule interface and add a app-name queue mapping rule as an > example > -- > > Key: YARN-8016 > URL: https://issues.apache.org/jira/browse/YARN-8016 > Project: Hadoop YARN > Issue Type: Task >Reporter: Zian Chen >Assignee: Zian Chen >Priority: Major > Attachments: YARN-8016.001.patch, YARN-8016.002.patch > > > After YARN-3635/YARN-6689, PlacementRule becomes a common interface which can > be used by scheduler and can be dynamically updated by scheduler according to > configs. There're some other works. > - There's no way to initialize PlacementRule. > - No example of PlacementRule except the user-group mapping one. > This JIRA is targeted to refine PlacementRule interfaces and add another > PlacementRule example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
