[jira] [Commented] (YARN-7850) New UI does not show status for Log Aggregation
[ https://issues.apache.org/jira/browse/YARN-7850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352150#comment-16352150 ] Sunil G commented on YARN-7850: --- Cool. Looks fine to me. I ll commit this later today. > New UI does not show status for Log Aggregation > --- > > Key: YARN-7850 > URL: https://issues.apache.org/jira/browse/YARN-7850 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Yesha Vora >Assignee: Gergely Novák >Priority: Major > Attachments: Screen Shot 2018-02-01 at 11.37.30.png, > YARN-7850.001.patch > > > The status of Log Aggregation is not specified any where. > New UI should show the Log aggregation status for finished application. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7893) Document the FPGA isolation feature
Zhankun Tang created YARN-7893: -- Summary: Document the FPGA isolation feature Key: YARN-7893 URL: https://issues.apache.org/jira/browse/YARN-7893 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhankun Tang -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5028) RMStateStore should trim down app state for completed applications
[
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergo Repas updated YARN-5028:
--
Attachment: YARN-5028.002.patch
> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch,
> YARN-5028.002.patch
>
>
> RMStateStore stores enough information to recover applications in case of a
> restart. The store also retains this information for completed applications
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all
> the information we store today to serve application status; for instance, we
> don't need the {{ApplicationSubmissionContext}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7346) Fix compilation errors against hbase2 beta release
[ https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352187#comment-16352187 ] Rohith Sharma K S commented on YARN-7346: - Update : As we discussed in last ATSv2 weekly call([~haibochen] [~vrushalic] [~varun_saxena]), decided to support conditional compilation so that ATSv2 should support hbase-1.2.6 which is already release along with Hadoop GA and HBase-2 as well. Currently all the hbase code are embedded in single jar. To support conditional compilation, straight forward approach is create another module which picks up if Hbase version is 2.0. But issue with this approach is duplicated code exist on both modules. After further discussion, we also thought of splitting into multiple jars such as *hadoop-yarn-server-timelineservice-hbase-common.jar* *hadoop-yarn-server-timelineservice-hbase-reader.jar* *hadoop-yarn-server-timelineservice-hbase-collector.jar* so that HBase co-processor code resides on collector jar which has co-processor modifications. Doing this brings out we can create any number of module for collectors if HBase changes any APIs. But cons from this approach is how does existing upgrade is handled? We also need to analyze running HBase with older timelineservice co processor jar and collector running with newer jars causes any issue provided we still maintain same class names. [~haibochen] [~vrushalic] feel free to add more point if I have missed any. > Fix compilation errors against hbase2 beta release > -- > > Key: YARN-7346 > URL: https://issues.apache.org/jira/browse/YARN-7346 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Vrushali C >Priority: Major > Attachments: YARN-7346.00.patch, YARN-7346.01.patch, > YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch > > > When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, > I got the following errors: > https://pastebin.com/Ms4jYEVB > This issue is to fix the compilation errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7894) Improve ATS response for DS_CONTAINER when container launch fails
Charan Hebri created YARN-7894:
--
Summary: Improve ATS response for DS_CONTAINER when container
launch fails
Key: YARN-7894
URL: https://issues.apache.org/jira/browse/YARN-7894
Project: Hadoop YARN
Issue Type: Bug
Components: ATSv2
Reporter: Charan Hebri
When a distributed shell application starts running and a container launch
fails the web service call to the API,
{noformat}
http:///ws/v1/timeline/DS_CONTAINER/{noformat}
return a "Not Found". The message returned in this case should be improved to
signify that a container launch failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7835) [Atsv2] Race condition in NM while publishing events if second attempt launched on same node
[ https://issues.apache.org/jira/browse/YARN-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352238#comment-16352238 ] Rohith Sharma K S commented on YARN-7835: - bq. An alternative would be be to only clean up the collector when the application finishes instead of when an AM container finishes It is doable and should be fine! One concern from very very rare scenario is this will make collector map to retain as long as application_stop event triggers. Lets take example where 1st attempt is running in Node-1 and killed. 2nd attempt started on different node, but Node-1 doesn't get application_stop event since application is still running which causes Node-1 to keep this map. Once application is finished, this will be removed but if it is long running application, then this map will retain in two nodemanagers. It would be become a gradual leak in case of long running applications. > [Atsv2] Race condition in NM while publishing events if second attempt > launched on same node > > > Key: YARN-7835 > URL: https://issues.apache.org/jira/browse/YARN-7835 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Critical > Attachments: YARN-7835.001.patch > > > It is observed race condition that if master container is killed for some > reason and launched on same node then NMTimelinePublisher doesn't add > timelineClient. But once completed container for 1st attempt has come then > NMTimelinePublisher removes the timelineClient. > It causes all subsequent event publishing from different client fails to > publish with exception Application is not found. ! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7859) New feature: add queue scheduling deadLine in fairScheduler.
[ https://issues.apache.org/jira/browse/YARN-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352252#comment-16352252 ] wangwj commented on YARN-7859: -- Hi,[~yufeigu].[~ka...@cloudera.com]..Thank you for your reply. Your example is roughly the same as the problems I encountered in my cluster. There are two and only two queues under the root, q1 and q2. q1 has many long-running large job and occupy large resources.Based on this,root.q1won't be scheduled for a while even there are available resources. So, I propose this issue...Add queue scheduling deadLine to improve this phenomenon.The implementation in my patch is a common solution like scheduling in disk and database. I have see the global scheduling! [YARN-5139|https://issues.apache.org/jira/browse/YARN-5139]But we don’t konw when we can use it.I think add queue scheduling deadLine is the most efficient way. Please give me some advices.. Thanks... > New feature: add queue scheduling deadLine in fairScheduler. > > > Key: YARN-7859 > URL: https://issues.apache.org/jira/browse/YARN-7859 > Project: Hadoop YARN > Issue Type: New Feature > Components: fairscheduler >Affects Versions: 3.0.0 >Reporter: wangwj >Assignee: wangwj >Priority: Major > Labels: fairscheduler, features, patch > Fix For: 3.0.0 > > Attachments: YARN-7859-v1.patch, log, screenshot-1.png, > screenshot-3.png > > Original Estimate: 24h > Remaining Estimate: 24h > > As everyone knows.In FairScheduler the phenomenon of queue scheduling > starvation often occurs when the number of cluster jobs is large.The App in > one or more queue are pending.So I have thought a way to solve this > problem.Add queue scheduling deadLine in fairScheduler.When a queue is not > scheduled for FairScheduler within a specified time.We mandatory scheduler it! > On the basis of the above, I propose this issue... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352287#comment-16352287
]
genericqa commented on YARN-5028:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 54s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 48s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-5028 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909200/YARN-5028.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux bfabe6b18699 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4e9a59c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19600/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19600/testReport/ |
| Max. process+thread count | 880 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/ha
[jira] [Commented] (YARN-5201) Apache Ranger Yarn policies are not used
[ https://issues.apache.org/jira/browse/YARN-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352338#comment-16352338 ] Sachin Gupta commented on YARN-5201: Is this really an issue? if no, could anyone please help me with a solution because I am facing the same issue > Apache Ranger Yarn policies are not used > > > Key: YARN-5201 > URL: https://issues.apache.org/jira/browse/YARN-5201 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rajendranath Rengan >Priority: Major > > Hi, > I have setup Apache Ranger in hadoop cluster and defined yarn policies to > allow certain user to certain queue. > Idea is to have user 'x' submit spark job only to queue 'x' and not to queue > 'y'. when submitting spark job queue is mentioned as one of the arguments > But user 'x' is able to submit spark job to queue 'y' > Ranger audit logs shows the policy used is HDFS policy > Yarn policy is not used at all. > I have enabled ranger plugin for YARN and defined yarn policy > Yarn ACL is also set to true > capacity scheduler setting as below: > yarn.scheduler.capacity.queue-mappings=u:user1:user1,u:user2:userr2 > yarn.scheduler.capacity.root.acl_submit_applications=yarn,spark,hdfs > yarn.scheduler.capacity.root.customer1.acl_administer_jobs=user1 > yarn.scheduler.capacity.root.customer1.acl_submit_applications=user1 > yarn.scheduler.capacity.root.customer1.capacity=50 > yarn.scheduler.capacity.root.customer1.maximum-capacity=100 > yarn.scheduler.capacity.root.customer1.state=RUNNING > yarn.scheduler.capacity.root.customer1.user-limit-factor=1 > yarn.scheduler.capacity.root.customer2.acl_administer_jobs=user2 > yarn.scheduler.capacity.root.customer2.acl_submit_applications=user2 > yarn.scheduler.capacity.root.customer2.capacity=50 > yarn.scheduler.capacity.root.customer2.maximum-capacity=100 > yarn.scheduler.capacity.root.customer2.state=RUNNING > yarn.scheduler.capacity.root.customer2.user-limit-factor=1 > yarn.scheduler.capacity.root.queues=user1,user2 > Thanks > Rengan -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6739) Crash NM at start time if oversubscription is on but LinuxContainerExcutor or cgroup is off
[ https://issues.apache.org/jira/browse/YARN-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6739: - Attachment: YARN-6739-YARN-1011.01.patch > Crash NM at start time if oversubscription is on but LinuxContainerExcutor or > cgroup is off > --- > > Key: YARN-6739 > URL: https://issues.apache.org/jira/browse/YARN-6739 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-6739-YARN-1011.00.patch, > YARN-6739-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6739) Crash NM at start time if oversubscription is on but LinuxContainerExcutor or cgroup is off
[ https://issues.apache.org/jira/browse/YARN-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352443#comment-16352443 ] Haibo Chen commented on YARN-6739: -- Thanks [~miklos.szeg...@cloudera.com] for the catch! I updated the patch based on your comment. > Crash NM at start time if oversubscription is on but LinuxContainerExcutor or > cgroup is off > --- > > Key: YARN-6739 > URL: https://issues.apache.org/jira/browse/YARN-6739 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-6739-YARN-1011.00.patch, > YARN-6739-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7890) NPE during container relaunch
[
https://issues.apache.org/jira/browse/YARN-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352503#comment-16352503
]
Jason Lowe commented on YARN-7890:
--
The NPE occurs because ContainerRelaunch never initializes the filecache
directories in the container start context builder. Looks like YARN-5298 added
new fields to the container start context but forgot to update
ContainerRelaunch to specify them.
> NPE during container relaunch
> -
>
> Key: YARN-7890
> URL: https://issues.apache.org/jira/browse/YARN-7890
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Billie Rinaldi
>Priority: Major
>
> While running a recent build of trunk, I saw the following:
> {noformat}
> 2018-02-02 21:02:40,026 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_e02_1517604848419_0002_01_04 transitioned from
> RELAUNCHING to RUNNING
> 2018-02-02 21:02:40,026 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch:
> Failed to relaunch container.
> java.lang.NullPointerException
> at
> java.util.Collections$UnmodifiableCollection.(Collections.java:1026)
> at
> java.util.Collections$UnmodifiableList.(Collections.java:1302)
> at java.util.Collections.unmodifiableList(Collections.java:1287)
> at
> org.apache.hadoop.yarn.server.nodemanager.executor.ContainerStartContext.getFilecacheDirs(ContainerStartContext.java:200)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.buildContainerRuntimeContext(LinuxContainerExecutor.java:651)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:546)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:465)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:49)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7893) Document the FPGA isolation feature
[ https://issues.apache.org/jira/browse/YARN-7893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-7893: --- Attachment: FPGA-doc-YARN-7893.pdf > Document the FPGA isolation feature > --- > > Key: YARN-7893 > URL: https://issues.apache.org/jira/browse/YARN-7893 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Priority: Major > Attachments: FPGA-doc-YARN-7893.pdf > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7893) Document the FPGA isolation feature
[ https://issues.apache.org/jira/browse/YARN-7893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-7893: --- Attachment: YARN-7893-trunk-001.patch > Document the FPGA isolation feature > --- > > Key: YARN-7893 > URL: https://issues.apache.org/jira/browse/YARN-7893 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Priority: Major > Attachments: FPGA-doc-YARN-7893.pdf, YARN-7893-trunk-001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7893) Document the FPGA isolation feature
[ https://issues.apache.org/jira/browse/YARN-7893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352548#comment-16352548 ] Zhankun Tang commented on YARN-7893: [~wangda] , [~zyluo] The draft doc is attached. Please review. > Document the FPGA isolation feature > --- > > Key: YARN-7893 > URL: https://issues.apache.org/jira/browse/YARN-7893 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Priority: Major > Attachments: FPGA-doc-YARN-7893.pdf, YARN-7893-trunk-001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6739) Crash NM at start time if oversubscription is on but LinuxContainerExcutor or cgroup is off
[
https://issues.apache.org/jira/browse/YARN-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352550#comment-16352550
]
genericqa commented on YARN-6739:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
30s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
2s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-6739 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909230/YARN-6739-YARN-1011.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 32d3357d74d3 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-1011 / 13d61cc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19601/testReport/ |
| Max. process+thread count | 407 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19601/console |
|
[jira] [Commented] (YARN-6507) Add support in NodeManager to isolate FPGA devices with CGroups
[ https://issues.apache.org/jira/browse/YARN-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352556#comment-16352556 ] Zhankun Tang commented on YARN-6507: [~tangzhankun] , A minor issue found in IntelOpenclFPGA when YARN user doesn't adopt needed environment variable and leave the execute binary path configuration empty. This will cause NM failure. Mark it here and will fix later. > Add support in NodeManager to isolate FPGA devices with CGroups > --- > > Key: YARN-6507 > URL: https://issues.apache.org/jira/browse/YARN-6507 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-6507-branch-YARN-3926.001.patch, > YARN-6507-branch-YARN-3926.002.patch, YARN-6507-trunk.001.patch, > YARN-6507-trunk.002.patch, YARN-6507-trunk.003.patch, > YARN-6507-trunk.004.patch, YARN-6507-trunk.005.patch, > YARN-6507-trunk.006.patch, YARN-6507-trunk.007.patch, > YARN-6507-trunk.008.patch, YARN-6507-trunk.009.patch, > YARN-6507-trunk.010.patch, YARN-6507-trunk.011.patch, > YARN-6507-trunk.012.patch > > > Support local FPGA resource scheduler to assign/isolate N FPGA slots to a > container. > At the beginning, support one vendor plugin with basic features to serve > OpenCL applications -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7890) NPE during container relaunch
[
https://issues.apache.org/jira/browse/YARN-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated YARN-7890:
-
Attachment: YARN-7890.001.patch
> NPE during container relaunch
> -
>
> Key: YARN-7890
> URL: https://issues.apache.org/jira/browse/YARN-7890
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Billie Rinaldi
>Priority: Major
> Attachments: YARN-7890.001.patch
>
>
> While running a recent build of trunk, I saw the following:
> {noformat}
> 2018-02-02 21:02:40,026 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_e02_1517604848419_0002_01_04 transitioned from
> RELAUNCHING to RUNNING
> 2018-02-02 21:02:40,026 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch:
> Failed to relaunch container.
> java.lang.NullPointerException
> at
> java.util.Collections$UnmodifiableCollection.(Collections.java:1026)
> at
> java.util.Collections$UnmodifiableList.(Collections.java:1302)
> at java.util.Collections.unmodifiableList(Collections.java:1287)
> at
> org.apache.hadoop.yarn.server.nodemanager.executor.ContainerStartContext.getFilecacheDirs(ContainerStartContext.java:200)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.buildContainerRuntimeContext(LinuxContainerExecutor.java:651)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:546)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:465)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:49)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7890) NPE during container relaunch
[
https://issues.apache.org/jira/browse/YARN-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe reassigned YARN-7890:
Assignee: Jason Lowe
Affects Version/s: 2.9.0
3.0.0-alpha1
Target Version/s: 3.1.0, 2.9.1, 3.0.1
Attached a patch that initializes the filecache and user local directories in
the relaunch container start context along with a unit test.
> NPE during container relaunch
> -
>
> Key: YARN-7890
> URL: https://issues.apache.org/jira/browse/YARN-7890
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1, 2.9.0
>Reporter: Billie Rinaldi
>Assignee: Jason Lowe
>Priority: Major
> Attachments: YARN-7890.001.patch
>
>
> While running a recent build of trunk, I saw the following:
> {noformat}
> 2018-02-02 21:02:40,026 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_e02_1517604848419_0002_01_04 transitioned from
> RELAUNCHING to RUNNING
> 2018-02-02 21:02:40,026 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch:
> Failed to relaunch container.
> java.lang.NullPointerException
> at
> java.util.Collections$UnmodifiableCollection.(Collections.java:1026)
> at
> java.util.Collections$UnmodifiableList.(Collections.java:1302)
> at java.util.Collections.unmodifiableList(Collections.java:1287)
> at
> org.apache.hadoop.yarn.server.nodemanager.executor.ContainerStartContext.getFilecacheDirs(ContainerStartContext.java:200)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.buildContainerRuntimeContext(LinuxContainerExecutor.java:651)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:546)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:465)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:49)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7894) Improve ATS response for DS_CONTAINER when container launch fails
[
https://issues.apache.org/jira/browse/YARN-7894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charan Hebri updated YARN-7894:
---
Component/s: (was: ATSv2)
timelineserver
> Improve ATS response for DS_CONTAINER when container launch fails
> -
>
> Key: YARN-7894
> URL: https://issues.apache.org/jira/browse/YARN-7894
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Charan Hebri
>Priority: Major
>
> When a distributed shell application starts running and a container launch
> fails the web service call to the API,
> {noformat}
> http:// address>/ws/v1/timeline/DS_CONTAINER/{noformat}
> return a "Not Found". The message returned in this case should be improved to
> signify that a container launch failed.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7815) Mount the filecache as read-only in Docker containers
[ https://issues.apache.org/jira/browse/YARN-7815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352614#comment-16352614 ] Shane Kumpf commented on YARN-7815: --- Cancelling the patch until YARN-7890 is in as I overlooked ContainerRelaunch as well. > Mount the filecache as read-only in Docker containers > - > > Key: YARN-7815 > URL: https://issues.apache.org/jira/browse/YARN-7815 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Shane Kumpf >Assignee: Shane Kumpf >Priority: Major > Attachments: YARN-7815.001.patch, YARN-7815.002.patch > > > Currently, when using the Docker runtime, the filecache directories are > mounted read-write into the Docker containers. Read write access is not > necessary. We should make this more restrictive by changing that mount to > read-only. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7815) Revisit the
[ https://issues.apache.org/jira/browse/YARN-7815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Kumpf updated YARN-7815: -- Summary: Revisit the (was: Mount the filecache as read-only in Docker containers) > Revisit the > > > Key: YARN-7815 > URL: https://issues.apache.org/jira/browse/YARN-7815 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Shane Kumpf >Assignee: Shane Kumpf >Priority: Major > Attachments: YARN-7815.001.patch, YARN-7815.002.patch > > > Currently, when using the Docker runtime, the filecache directories are > mounted read-write into the Docker containers. Read write access is not > necessary. We should make this more restrictive by changing that mount to > read-only. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7815) Make the YARN mounts added to Docker containers more restrictive
[ https://issues.apache.org/jira/browse/YARN-7815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Kumpf updated YARN-7815: -- Summary: Make the YARN mounts added to Docker containers more restrictive (was: Revisit the ) > Make the YARN mounts added to Docker containers more restrictive > > > Key: YARN-7815 > URL: https://issues.apache.org/jira/browse/YARN-7815 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Shane Kumpf >Assignee: Shane Kumpf >Priority: Major > Attachments: YARN-7815.001.patch, YARN-7815.002.patch > > > Currently, when using the Docker runtime, the filecache directories are > mounted read-write into the Docker containers. Read write access is not > necessary. We should make this more restrictive by changing that mount to > read-only. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7890) NPE during container relaunch
[
https://issues.apache.org/jira/browse/YARN-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352683#comment-16352683
]
genericqa commented on YARN-7890:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 1 new + 2 unchanged - 1 fixed = 3 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
18s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7890 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909241/YARN-7890.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 67a54fa705ee 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4e9a59c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19602/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19602/testReport/ |
| Max. process+thread count | 302 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-
[jira] [Created] (YARN-7895) Backport YARN-3564 to branch-2.7 to fix TestContainerAllocation
Chen Liang created YARN-7895:
Summary: Backport YARN-3564 to branch-2.7 to fix
TestContainerAllocation
Key: YARN-7895
URL: https://issues.apache.org/jira/browse/YARN-7895
Project: Hadoop YARN
Issue Type: Bug
Components: test
Reporter: Chen Liang
We are hitting a consistently failing test
{{testAMContainerAllocationWhenDNSUnavailable}} in {{TestContainerAllocation}}
on our branch-2.7. Tested YARN-3564 locally, which looks to be fixing this
test. Filing this jira to backport to branch-2.7.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7895) Backport YARN-3564 to branch-2.7 to fix TestContainerAllocation
[
https://issues.apache.org/jira/browse/YARN-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated YARN-7895:
-
Attachment: YARN-7895-branch-2.7.diff
> Backport YARN-3564 to branch-2.7 to fix TestContainerAllocation
> ---
>
> Key: YARN-7895
> URL: https://issues.apache.org/jira/browse/YARN-7895
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Chen Liang
>Priority: Minor
> Attachments: YARN-7895-branch-2.7.diff
>
>
> We are hitting a consistently failing test
> {{testAMContainerAllocationWhenDNSUnavailable}} in
> {{TestContainerAllocation}} on our branch-2.7. Tested YARN-3564 locally,
> which looks to be fixing this test. Filing this jira to backport to
> branch-2.7.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7895) Backport YARN-3564 to branch-2.7 to fix TestContainerAllocation
[
https://issues.apache.org/jira/browse/YARN-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated YARN-7895:
-
Description: We are hitting a consistently failing test
{{testAMContainerAllocationWhenDNSUnavailable}} in {{TestContainerAllocation}}
in our internal branch-2.7. Tested YARN-3564 locally, which looks to be fixing
this test. Filing this jira to backport YARN-3564 to branch-2.7. (was: We are
hitting a consistently failing test
{{testAMContainerAllocationWhenDNSUnavailable}} in {{TestContainerAllocation}}
on our branch-2.7. Tested YARN-3564 locally, which looks to be fixing this
test. Filing this jira to backport to branch-2.7.)
> Backport YARN-3564 to branch-2.7 to fix TestContainerAllocation
> ---
>
> Key: YARN-7895
> URL: https://issues.apache.org/jira/browse/YARN-7895
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Chen Liang
>Priority: Minor
> Attachments: YARN-7895-branch-2.7.diff
>
>
> We are hitting a consistently failing test
> {{testAMContainerAllocationWhenDNSUnavailable}} in
> {{TestContainerAllocation}} in our internal branch-2.7. Tested YARN-3564
> locally, which looks to be fixing this test. Filing this jira to backport
> YARN-3564 to branch-2.7.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7895) Backport YARN-3564 to branch-2.7 to fix TestContainerAllocation
[
https://issues.apache.org/jira/browse/YARN-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated YARN-7895:
-
Attachment: YARN-7895-branch-2.7.patch
> Backport YARN-3564 to branch-2.7 to fix TestContainerAllocation
> ---
>
> Key: YARN-7895
> URL: https://issues.apache.org/jira/browse/YARN-7895
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Chen Liang
>Priority: Minor
> Attachments: YARN-7895-branch-2.7.patch
>
>
> We are hitting a consistently failing test
> {{testAMContainerAllocationWhenDNSUnavailable}} in
> {{TestContainerAllocation}} in our internal branch-2.7. Tested YARN-3564
> locally, which looks to be fixing this test. Filing this jira to backport
> YARN-3564 to branch-2.7.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7895) Backport YARN-3564 to branch-2.7 to fix TestContainerAllocation
[
https://issues.apache.org/jira/browse/YARN-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated YARN-7895:
-
Attachment: (was: YARN-7895-branch-2.7.diff)
> Backport YARN-3564 to branch-2.7 to fix TestContainerAllocation
> ---
>
> Key: YARN-7895
> URL: https://issues.apache.org/jira/browse/YARN-7895
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Chen Liang
>Priority: Minor
> Attachments: YARN-7895-branch-2.7.patch
>
>
> We are hitting a consistently failing test
> {{testAMContainerAllocationWhenDNSUnavailable}} in
> {{TestContainerAllocation}} in our internal branch-2.7. Tested YARN-3564
> locally, which looks to be fixing this test. Filing this jira to backport
> YARN-3564 to branch-2.7.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7895) Backport YARN-3564 to branch-2.7 to fix TestContainerAllocation
[
https://issues.apache.org/jira/browse/YARN-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352720#comment-16352720
]
Chen Liang commented on YARN-7895:
--
Posting a patch (not able to assign the JIRA though). [~jianhe] could you
please take a look?
> Backport YARN-3564 to branch-2.7 to fix TestContainerAllocation
> ---
>
> Key: YARN-7895
> URL: https://issues.apache.org/jira/browse/YARN-7895
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Chen Liang
>Priority: Minor
> Attachments: YARN-7895-branch-2.7.patch
>
>
> We are hitting a consistently failing test
> {{testAMContainerAllocationWhenDNSUnavailable}} in
> {{TestContainerAllocation}} in our internal branch-2.7. Tested YARN-3564
> locally, which looks to be fixing this test. Filing this jira to backport
> YARN-3564 to branch-2.7.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-6868) Add test scope to certain entries in hadoop-yarn-server-resourcemanager pom.xml
[
https://issues.apache.org/jira/browse/YARN-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas reopened YARN-6868:
-
Sure. Reopening to cherry-pick this to branch-2 and branch-2.9
> Add test scope to certain entries in hadoop-yarn-server-resourcemanager
> pom.xml
> ---
>
> Key: YARN-6868
> URL: https://issues.apache.org/jira/browse/YARN-6868
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6868.001.patch
>
>
> The tag
> {noformat}
> test
> {noformat}
> is missing from a few entries in the pom.xml for
> hadoop-yarn-server-resourcemanager.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6868) Add test scope to certain entries in hadoop-yarn-server-resourcemanager pom.xml
[
https://issues.apache.org/jira/browse/YARN-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas resolved YARN-6868.
-
Resolution: Fixed
Fix Version/s: (was: 3.0.0-beta1)
2.9.1
> Add test scope to certain entries in hadoop-yarn-server-resourcemanager
> pom.xml
> ---
>
> Key: YARN-6868
> URL: https://issues.apache.org/jira/browse/YARN-6868
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 2.9.1
>
> Attachments: YARN-6868.001.patch
>
>
> The tag
> {noformat}
> test
> {noformat}
> is missing from a few entries in the pom.xml for
> hadoop-yarn-server-resourcemanager.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6868) Add test scope to certain entries in hadoop-yarn-server-resourcemanager pom.xml
[
https://issues.apache.org/jira/browse/YARN-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352774#comment-16352774
]
Chris Douglas commented on YARN-6868:
-
Verified that the zookeeper and curator test jars aren't part of the package
after backporting, pushed.
> Add test scope to certain entries in hadoop-yarn-server-resourcemanager
> pom.xml
> ---
>
> Key: YARN-6868
> URL: https://issues.apache.org/jira/browse/YARN-6868
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 2.9.1
>
> Attachments: YARN-6868.001.patch
>
>
> The tag
> {noformat}
> test
> {noformat}
> is missing from a few entries in the pom.xml for
> hadoop-yarn-server-resourcemanager.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7876) Localized jars that are expanded after localization are not fully copied
[ https://issues.apache.org/jira/browse/YARN-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352778#comment-16352778 ] Hudson commented on YARN-7876: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13613 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13613/]) YARN-7876. Localized jars that are expanded after localization are not (jlowe: rev 33e6cdbd16baca512a7e25de2b8e253cab07dce3) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/RunJar.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestRunJar.java > Localized jars that are expanded after localization are not fully copied > > > Key: YARN-7876 > URL: https://issues.apache.org/jira/browse/YARN-7876 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Blocker > Fix For: 3.1.0 > > Attachments: YARN-7876.000.patch, YARN-7876.001.patch > > > YARN-2185 added the ability to localize jar files as a stream instead of > copying to local disk and then extracting. ZipInputStream does not need the > end of the file. Let's read it out. This helps with an additional > TeeInputStream on the input. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7857) -fstack-check compilation flag causes binary incompatibility for container-executor between RHEL 6 and RHEL 7
[
https://issues.apache.org/jira/browse/YARN-7857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352816#comment-16352816
]
Jim Brennan commented on YARN-7857:
---
[~miklos.szeg...@cloudera.com] I have prepared another patch that only adds the
{{-fstack-check}} option for GCC versions > 4.8.
But on further review and reflection, I have come around to the opinion that
the security issue is more important than this incompatibility, especially
given that we have addressed that incompatibility in YARN-7796.
I have not found anything that officially changes the recommendation of using
{{-fstack-check}} to help combat stack clash attacks, and I have not found an
alternative command line option for gcc. So I am reluctant to remove
{{-fstack-check}} for any versions of GCC when it is currently not causing a
problem. My original motivation of preventing us from running into the same
incompatibility again (due to future changes to container-executor code) does
not seem worth re-opening a significant security hole.
> -fstack-check compilation flag causes binary incompatibility for
> container-executor between RHEL 6 and RHEL 7
> -
>
> Key: YARN-7857
> URL: https://issues.apache.org/jira/browse/YARN-7857
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-7857.001.patch
>
>
> The segmentation fault in container-executor reported in [YARN-7796] appears
> to be due to a binary compatibility issue with the {{-fstack-check}} flag
> that was added in [YARN-6721]
> Based on my testing, a container-executor (without the patch from
> [YARN-7796]) compiled on RHEL 6 with the -fstack-check flag always hits this
> segmentation fault when run on RHEL 7. But if you compile without this flag,
> the container-executor runs on RHEL 7 with no problems. I also verified this
> with a simple program that just does the copy_file.
> I think we need to either remove this flag, or find a suitable alternative.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7857) -fstack-check compilation flag causes binary incompatibility for container-executor between RHEL 6 and RHEL 7
[
https://issues.apache.org/jira/browse/YARN-7857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352824#comment-16352824
]
Miklos Szegedi commented on YARN-7857:
--
[~Jim_Brennan], I agree let's postpone removing the guard. The RH7 code checks
much less. It seems like it checks only the pages that it needs, so probably
that is why it is not crashing like the RH6 code. However, I am interested why
the kernel traps the expansion of the stack despite of the fact that we are
within the actual limit of the stack.
> -fstack-check compilation flag causes binary incompatibility for
> container-executor between RHEL 6 and RHEL 7
> -
>
> Key: YARN-7857
> URL: https://issues.apache.org/jira/browse/YARN-7857
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-7857.001.patch
>
>
> The segmentation fault in container-executor reported in [YARN-7796] appears
> to be due to a binary compatibility issue with the {{-fstack-check}} flag
> that was added in [YARN-6721]
> Based on my testing, a container-executor (without the patch from
> [YARN-7796]) compiled on RHEL 6 with the -fstack-check flag always hits this
> segmentation fault when run on RHEL 7. But if you compile without this flag,
> the container-executor runs on RHEL 7 with no problems. I also verified this
> with a simple program that just does the copy_file.
> I think we need to either remove this flag, or find a suitable alternative.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7849) TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error
[
https://issues.apache.org/jira/browse/YARN-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Botong Huang updated YARN-7849:
---
Attachment: YARN-7849.v2.patch
> TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to
> heartbeat sync error
> --
>
> Key: YARN-7849
> URL: https://issues.apache.org/jira/browse/YARN-7849
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.1.0, 2.9.1, 3.0.1, 2.8.4
>Reporter: Jason Lowe
>Assignee: Botong Huang
>Priority: Major
> Attachments: YARN-7849-branch-2.8.v1.patch, YARN-7849.v1.patch,
> YARN-7849.v2.patch
>
>
> testUpdateNodeUtilization is failing. From a branch-2.8 run:
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 13.013 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 12.961 sec <<< FAILURE!
> java.lang.AssertionError: Containers Utillization not propagated to RMNode
> expected:<> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:227)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7655) avoid AM preemption caused by RRs for specific nodes or racks
[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352866#comment-16352866
]
Yufei Gu commented on YARN-7655:
Hi [~Steven Rand], I made it work by using following resource requests:
{code}
ResourceRequest nodeRequest =
createResourceRequest(GB, node1.getHostName(), 1, 4, true);
ResourceRequest rackRequest =
createResourceRequest(GB, node1.getRackName(), 1, 4, true);
ResourceRequest anyRequest =
createResourceRequest(GB, ResourceRequest.ANY, 1, 4, true);
...
verifyPreemption(4, 4);
{code}
Does this sound good to you?
> avoid AM preemption caused by RRs for specific nodes or racks
> -
>
> Key: YARN-7655
> URL: https://issues.apache.org/jira/browse/YARN-7655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Attachments: YARN-7655-001.patch, YARN-7655-002.patch
>
>
> We frequently see AM preemptions when
> {{starvedApp.getStarvedResourceRequests()}} in
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs
> that request containers on a specific node. Since this causes us to only
> consider one node to preempt containers on, the really good work that was
> done in YARN-5830 doesn't save us from AM preemption. Even though there might
> be multiple nodes on which we could preempt enough non-AM containers to
> satisfy the app's starvation, we often wind up preempting one or more AM
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM
> containers for an RR that specifies a node or rack, then we should instead
> expand the search space to consider all nodes. That way we take advantage of
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a
> patch with an initial implementation of this. We've been running it on a few
> clusters, and have seen AM preemptions drop from double-digit occurrences on
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is
> less likely to be allocated resources at the most specific locality level
> that it asked for. My opinion is that this tradeoff is worth it, but
> interested to hear what others think as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7655) avoid AM preemption caused by RRs for specific nodes or racks
[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352866#comment-16352866
]
Yufei Gu edited comment on YARN-7655 at 2/5/18 8:11 PM:
Hi [~Steven Rand], I made it work by using following resource requests:
{code}
ResourceRequest nodeRequest =
createResourceRequest(GB, node1.getHostName(), 1, 4, true);
ResourceRequest rackRequest =
createResourceRequest(GB, node1.getRackName(), 1, 4, true);
ResourceRequest anyRequest =
createResourceRequest(GB, ResourceRequest.ANY, 1, 4, true);
...
verifyPreemption(4, 4);
{code}
Does this sound good to you?
BTW, there is one extra empty line after method setAllAMContainersOnNode.
was (Author: yufeigu):
Hi [~Steven Rand], I made it work by using following resource requests:
{code}
ResourceRequest nodeRequest =
createResourceRequest(GB, node1.getHostName(), 1, 4, true);
ResourceRequest rackRequest =
createResourceRequest(GB, node1.getRackName(), 1, 4, true);
ResourceRequest anyRequest =
createResourceRequest(GB, ResourceRequest.ANY, 1, 4, true);
...
verifyPreemption(4, 4);
{code}
Does this sound good to you?
> avoid AM preemption caused by RRs for specific nodes or racks
> -
>
> Key: YARN-7655
> URL: https://issues.apache.org/jira/browse/YARN-7655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Attachments: YARN-7655-001.patch, YARN-7655-002.patch
>
>
> We frequently see AM preemptions when
> {{starvedApp.getStarvedResourceRequests()}} in
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs
> that request containers on a specific node. Since this causes us to only
> consider one node to preempt containers on, the really good work that was
> done in YARN-5830 doesn't save us from AM preemption. Even though there might
> be multiple nodes on which we could preempt enough non-AM containers to
> satisfy the app's starvation, we often wind up preempting one or more AM
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM
> containers for an RR that specifies a node or rack, then we should instead
> expand the search space to consider all nodes. That way we take advantage of
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a
> patch with an initial implementation of this. We've been running it on a few
> clusters, and have seen AM preemptions drop from double-digit occurrences on
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is
> less likely to be allocated resources at the most specific locality level
> that it asked for. My opinion is that this tradeoff is worth it, but
> interested to hear what others think as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7572) Make the service status output more readable
[
https://issues.apache.org/jira/browse/YARN-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352868#comment-16352868
]
Chandni Singh commented on YARN-7572:
-
[~eyang] we do print out pre-formatted JSON currently.
The current output of service status is
{code:java}
{
"uri" : null,
"name" : "test2",
"id" : "application_1517426742923_0001",
"artifact" : null,
"resource" : null,
"launch_time" : null,
"number_of_running_containers" : null,
"lifetime" : -1,
"placement_policy" : null,
"components" : [ {
"name" : "sleeper",
"dependencies" : [ ],
"readiness_check" : null,
"artifact" : null,
"launch_command" : "sleep 90",
"resource" : {
"uri" : null,
"profile" : null,
"cpus" : 1,
"memory" : "256",
"additional" : null
},
"number_of_containers" : 2,
"run_privileged_container" : false,
"placement_policy" : null,
"state" : "FLEXING",
"configuration" : {
"properties" : { },
"env" : { },
"files" : [ ]
},
"quicklinks" : [ ],
"containers" : [ ]
} ],
"configuration" : {
"properties" : { },
"env" : { },
"files" : [ ]
},
"state" : "STARTED",
"quicklinks" : { },
"queue" : null,
"kerberos_principal" : {
"principal_name" : null,
"keytab" : null
}
}{code}
> Make the service status output more readable
> -
>
> Key: YARN-7572
> URL: https://issues.apache.org/jira/browse/YARN-7572
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Chandni Singh
>Priority: Major
> Fix For: yarn-native-services
>
>
> Currently the service status output is just a JSON spec, we can make it more
> human readable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7841) Cleanup AllocationFileLoaderService's reloadAllocations method
[
https://issues.apache.org/jira/browse/YARN-7841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352896#comment-16352896
]
Robert Kanter commented on YARN-7841:
-
Thanks for the patch [~snemeth] and the review [~grepas]. Overall looks good.
A few minor things:
- Some files are using {{*}} imports. We should change those to explicit
imports.
- No need to bother with the 80 character limit for import lines
- Take a look through the [checkstyle
warnings|https://builds.apache.org/job/PreCommit-YARN-Build/19512/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt].
There's some valid ones in here like line lengths and missing javadoc
comments.
> Cleanup AllocationFileLoaderService's reloadAllocations method
> --
>
> Key: YARN-7841
> URL: https://issues.apache.org/jira/browse/YARN-7841
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.0.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-7841-001.patch, YARN-7841-002.patch
>
>
> AllocationFileLoaderService's reloadAllocations method is too complex.
> Please refactor / cleanup this method to be more simple to understand.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7859) New feature: add queue scheduling deadLine in fairScheduler.
[
https://issues.apache.org/jira/browse/YARN-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352921#comment-16352921
]
Yufei Gu commented on YARN-7859:
Basically, we should be very careful about introducing new properties to queue.
Take {{maxAMShare}} as an example. Community spends years to make it right.
There are always corner cases because of combination of multiple queue
properties. I am not sure whether it is worth to introduce one queue property
to solve one corner case which potentially introduce more corner cases.
Global scheduling is the right way to solve this kind of issue and on the todo
list of community. We could postpone this or thinking about alternatives. For
this particular issue, there is one easy way to mitigate, which is setting
minShare to a smaller value for long-running job queue. In that case, its
minShare usages are supposed to be larger and get more chance to be scheduled.
[~miklos.szeg...@cloudera.com], [~wilfreds], what's your take on this?
> New feature: add queue scheduling deadLine in fairScheduler.
>
>
> Key: YARN-7859
> URL: https://issues.apache.org/jira/browse/YARN-7859
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: wangwj
>Assignee: wangwj
>Priority: Major
> Labels: fairscheduler, features, patch
> Fix For: 3.0.0
>
> Attachments: YARN-7859-v1.patch, log, screenshot-1.png,
> screenshot-3.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> As everyone knows.In FairScheduler the phenomenon of queue scheduling
> starvation often occurs when the number of cluster jobs is large.The App in
> one or more queue are pending.So I have thought a way to solve this
> problem.Add queue scheduling deadLine in fairScheduler.When a queue is not
> scheduled for FairScheduler within a specified time.We mandatory scheduler it!
> On the basis of the above, I propose this issue...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7849) TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error
[
https://issues.apache.org/jira/browse/YARN-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352928#comment-16352928
]
genericqa commented on YARN-7849:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 8s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:
The patch generated 3 new + 33 unchanged - 0 fixed = 36 total (was 33) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
4s{color} | {color:green} hadoop-yarn-server-tests in the patch passed. {color}
|
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7849 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909268/YARN-7849.v2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d414eb5b03ca 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33e6cdb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| checkstyle |
htt
[jira] [Created] (YARN-7896) AM log link in diagnostic is redirected to old RM UI
Yesha Vora created YARN-7896:
Summary: AM log link in diagnostic is redirected to old RM UI
Key: YARN-7896
URL: https://issues.apache.org/jira/browse/YARN-7896
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-ui-v2
Reporter: Yesha Vora
Scenario:
1) Run Httpd yarn service in secure mode and make sure application gets
launched as dr.who
2) Go to Diagnostic tab
The message showed in Diagnostics mentions to AM UI link for old UI.
In diagnostic message UI2 link should be mentioned.
{code:java|title=Diagnostics}
Application application_1517253048795_0001 failed 20 times due to AM Container
for appattempt_1517253048795_0001_20 exited with exitCode: -1000 Failing
this attempt.Diagnostics: [2018-01-29 23:01:46.234]Application
application_1517253048795_0001 initialization failed (exitCode=255) with
output: main : command provided 0 main : run as user is dr.who main : requested
yarn user is dr.who User dr.who not found For more detailed output, check the
application tracking page:
http://xxx:8088/cluster/app/application_1517253048795_0001 Then click on links
to logs of each attempt. . Failing the application.{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7896) AM log link in diagnostic is redirected to old RM UI
[
https://issues.apache.org/jira/browse/YARN-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yesha Vora updated YARN-7896:
-
Affects Version/s: 3.0.0
> AM log link in diagnostic is redirected to old RM UI
>
>
> Key: YARN-7896
> URL: https://issues.apache.org/jira/browse/YARN-7896
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Priority: Major
>
> Scenario:
> 1) Run Httpd yarn service in secure mode and make sure application gets
> launched as dr.who
> 2) Go to Diagnostic tab
> The message showed in Diagnostics mentions to AM UI link for old UI.
> In diagnostic message UI2 link should be mentioned.
> {code:java|title=Diagnostics}
> Application application_1517253048795_0001 failed 20 times due to AM
> Container for appattempt_1517253048795_0001_20 exited with exitCode:
> -1000 Failing this attempt.Diagnostics: [2018-01-29 23:01:46.234]Application
> application_1517253048795_0001 initialization failed (exitCode=255) with
> output: main : command provided 0 main : run as user is dr.who main :
> requested yarn user is dr.who User dr.who not found For more detailed output,
> check the application tracking page:
> http://xxx:8088/cluster/app/application_1517253048795_0001 Then click on
> links to logs of each attempt. . Failing the application.{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7849) TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error
[
https://issues.apache.org/jira/browse/YARN-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Botong Huang updated YARN-7849:
---
Attachment: YARN-7849.v3.patch
> TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to
> heartbeat sync error
> --
>
> Key: YARN-7849
> URL: https://issues.apache.org/jira/browse/YARN-7849
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.1.0, 2.9.1, 3.0.1, 2.8.4
>Reporter: Jason Lowe
>Assignee: Botong Huang
>Priority: Major
> Attachments: YARN-7849-branch-2.8.v1.patch, YARN-7849.v1.patch,
> YARN-7849.v2.patch, YARN-7849.v3.patch
>
>
> testUpdateNodeUtilization is failing. From a branch-2.8 run:
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 13.013 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 12.961 sec <<< FAILURE!
> java.lang.AssertionError: Containers Utillization not propagated to RMNode
> expected:<> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:227)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7890) NPE during container relaunch
[
https://issues.apache.org/jira/browse/YARN-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352962#comment-16352962
]
Shane Kumpf commented on YARN-7890:
---
Thanks for the patch [~jlowe]. I've validated that it fixes the NPE. Looks like
there is a minor checkstyle issue. I'm +1 (non-binding) otherwise.
> NPE during container relaunch
> -
>
> Key: YARN-7890
> URL: https://issues.apache.org/jira/browse/YARN-7890
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Billie Rinaldi
>Assignee: Jason Lowe
>Priority: Major
> Attachments: YARN-7890.001.patch
>
>
> While running a recent build of trunk, I saw the following:
> {noformat}
> 2018-02-02 21:02:40,026 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_e02_1517604848419_0002_01_04 transitioned from
> RELAUNCHING to RUNNING
> 2018-02-02 21:02:40,026 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch:
> Failed to relaunch container.
> java.lang.NullPointerException
> at
> java.util.Collections$UnmodifiableCollection.(Collections.java:1026)
> at
> java.util.Collections$UnmodifiableList.(Collections.java:1302)
> at java.util.Collections.unmodifiableList(Collections.java:1287)
> at
> org.apache.hadoop.yarn.server.nodemanager.executor.ContainerStartContext.getFilecacheDirs(ContainerStartContext.java:200)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.buildContainerRuntimeContext(LinuxContainerExecutor.java:651)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:546)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:465)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:49)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7849) TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error
[
https://issues.apache.org/jira/browse/YARN-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352976#comment-16352976
]
genericqa commented on YARN-7849:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 5s{color}
| {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.TestDiskFailures |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7849 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909281/YARN-7849.v3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0a2683e1987c 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33e6cdb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19604/artifact/ou
[jira] [Commented] (YARN-7849) TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error
[
https://issues.apache.org/jira/browse/YARN-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352982#comment-16352982
]
Botong Huang commented on YARN-7849:
Hi [~jlowe], I think both v1 and v3 patch are good. Please let me know which
one you prefer. Thanks!
> TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to
> heartbeat sync error
> --
>
> Key: YARN-7849
> URL: https://issues.apache.org/jira/browse/YARN-7849
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.1.0, 2.9.1, 3.0.1, 2.8.4
>Reporter: Jason Lowe
>Assignee: Botong Huang
>Priority: Major
> Attachments: YARN-7849-branch-2.8.v1.patch, YARN-7849.v1.patch,
> YARN-7849.v2.patch, YARN-7849.v3.patch
>
>
> testUpdateNodeUtilization is failing. From a branch-2.8 run:
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 13.013 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 12.961 sec <<< FAILURE!
> java.lang.AssertionError: Containers Utillization not propagated to RMNode
> expected:<> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:227)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7657) Queue Mapping could provide options to provide 'user' specific auto-created queues under a specified group parent queue
[ https://issues.apache.org/jira/browse/YARN-7657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-7657: --- Attachment: YARN-7657.1.patch > Queue Mapping could provide options to provide 'user' specific auto-created > queues under a specified group parent queue > --- > > Key: YARN-7657 > URL: https://issues.apache.org/jira/browse/YARN-7657 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Attachments: YARN-7657.1.patch > > > Current Queue-Mapping only provides %user as an option for 'user' specific > queues as u:%user:%user. We can also support %user with group as > 'g:makerting-group:marketing.%user' and user specific queues can be > automatically created under a group queue in this case. > cc [~leftnoteasy] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7639) Queue Management scheduling edit policy class needs to be configured dynamically
[ https://issues.apache.org/jira/browse/YARN-7639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-7639: --- Description: This needs to be configured dynamically for yarn.resourcemanager.scheduler.monitor.policies whenever auto leaf queue creation is enabled for a parent queue. was:This needs to be configured dynamically for yarn.resourcemanager.monitor.capacity.queue-management.monitoring-interval whenever auto leaf queue creation is enabled for a parent queue. > Queue Management scheduling edit policy class needs to be configured > dynamically > > > Key: YARN-7639 > URL: https://issues.apache.org/jira/browse/YARN-7639 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > > This needs to be configured dynamically for > yarn.resourcemanager.scheduler.monitor.policies > whenever auto leaf queue creation is enabled for a parent queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7639) Queue Management scheduling edit policy class needs to be configured dynamically
[ https://issues.apache.org/jira/browse/YARN-7639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-7639: --- Description: This needs to be configured dynamically i.e added to the list of current policies configured under yarn.resourcemanager.scheduler.monitor.policies whenever auto leaf queue creation is enabled for a parent queue. was: This needs to be configured dynamically for yarn.resourcemanager.scheduler.monitor.policies whenever auto leaf queue creation is enabled for a parent queue. > Queue Management scheduling edit policy class needs to be configured > dynamically > > > Key: YARN-7639 > URL: https://issues.apache.org/jira/browse/YARN-7639 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > > This needs to be configured dynamically i.e added to the list of current > policies configured under > yarn.resourcemanager.scheduler.monitor.policies > whenever auto leaf queue creation is enabled for a parent queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7890) NPE during container relaunch
[
https://issues.apache.org/jira/browse/YARN-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated YARN-7890:
-
Attachment: YARN-7890.002.patch
> NPE during container relaunch
> -
>
> Key: YARN-7890
> URL: https://issues.apache.org/jira/browse/YARN-7890
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Billie Rinaldi
>Assignee: Jason Lowe
>Priority: Major
> Attachments: YARN-7890.001.patch, YARN-7890.002.patch
>
>
> While running a recent build of trunk, I saw the following:
> {noformat}
> 2018-02-02 21:02:40,026 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_e02_1517604848419_0002_01_04 transitioned from
> RELAUNCHING to RUNNING
> 2018-02-02 21:02:40,026 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch:
> Failed to relaunch container.
> java.lang.NullPointerException
> at
> java.util.Collections$UnmodifiableCollection.(Collections.java:1026)
> at
> java.util.Collections$UnmodifiableList.(Collections.java:1302)
> at java.util.Collections.unmodifiableList(Collections.java:1287)
> at
> org.apache.hadoop.yarn.server.nodemanager.executor.ContainerStartContext.getFilecacheDirs(ContainerStartContext.java:200)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.buildContainerRuntimeContext(LinuxContainerExecutor.java:651)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:546)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:465)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:49)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7890) NPE during container relaunch
[
https://issues.apache.org/jira/browse/YARN-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353007#comment-16353007
]
Jason Lowe commented on YARN-7890:
--
Thanks for the review, Shane! I added a token comment to make checkstyle happy
on the new unit test file.
> NPE during container relaunch
> -
>
> Key: YARN-7890
> URL: https://issues.apache.org/jira/browse/YARN-7890
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Billie Rinaldi
>Assignee: Jason Lowe
>Priority: Major
> Attachments: YARN-7890.001.patch, YARN-7890.002.patch
>
>
> While running a recent build of trunk, I saw the following:
> {noformat}
> 2018-02-02 21:02:40,026 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_e02_1517604848419_0002_01_04 transitioned from
> RELAUNCHING to RUNNING
> 2018-02-02 21:02:40,026 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch:
> Failed to relaunch container.
> java.lang.NullPointerException
> at
> java.util.Collections$UnmodifiableCollection.(Collections.java:1026)
> at
> java.util.Collections$UnmodifiableList.(Collections.java:1302)
> at java.util.Collections.unmodifiableList(Collections.java:1287)
> at
> org.apache.hadoop.yarn.server.nodemanager.executor.ContainerStartContext.getFilecacheDirs(ContainerStartContext.java:200)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.buildContainerRuntimeContext(LinuxContainerExecutor.java:651)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:546)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:465)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:49)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7876) Localized jars that are expanded after localization are not fully copied
[ https://issues.apache.org/jira/browse/YARN-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353017#comment-16353017 ] Miklos Szegedi commented on YARN-7876: -- Thank you for committing this [~jlowe]! > Localized jars that are expanded after localization are not fully copied > > > Key: YARN-7876 > URL: https://issues.apache.org/jira/browse/YARN-7876 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Blocker > Fix For: 3.1.0 > > Attachments: YARN-7876.000.patch, YARN-7876.001.patch > > > YARN-2185 added the ability to localize jar files as a stream instead of > copying to local disk and then extracting. ZipInputStream does not need the > end of the file. Let's read it out. This helps with an additional > TeeInputStream on the input. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7889) Missing kerberos token when check for RM REST API availability
[ https://issues.apache.org/jira/browse/YARN-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7889: Attachment: YARN-7889.002.patch > Missing kerberos token when check for RM REST API availability > -- > > Key: YARN-7889 > URL: https://issues.apache.org/jira/browse/YARN-7889 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-7889.001.patch, YARN-7889.002.patch > > > When checking for which resource manager can be used for REST API request, > client side must send kerberos token to REST API end point. The checking > mechanism is currently missing the kerberos token. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7849) TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error
[
https://issues.apache.org/jira/browse/YARN-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353077#comment-16353077
]
Jason Lowe commented on YARN-7849:
--
Thanks for the explanation, Botong! I see now how it should be OK to do the
out-of-band heartbeat. Convoluted, but it should still work. I think it's a
bit cleaner to avoid the async heartbeating in the tests, so I'd rather go with
the v3 patch.
The unit test failure is unrelated. +1 for the v3 patch, committing this.
> TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to
> heartbeat sync error
> --
>
> Key: YARN-7849
> URL: https://issues.apache.org/jira/browse/YARN-7849
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.1.0, 2.9.1, 3.0.1, 2.8.4
>Reporter: Jason Lowe
>Assignee: Botong Huang
>Priority: Major
> Attachments: YARN-7849-branch-2.8.v1.patch, YARN-7849.v1.patch,
> YARN-7849.v2.patch, YARN-7849.v3.patch
>
>
> testUpdateNodeUtilization is failing. From a branch-2.8 run:
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 13.013 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 12.961 sec <<< FAILURE!
> java.lang.AssertionError: Containers Utillization not propagated to RMNode
> expected:<> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:227)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7859) New feature: add queue scheduling deadLine in fairScheduler.
[
https://issues.apache.org/jira/browse/YARN-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353084#comment-16353084
]
Miklos Szegedi commented on YARN-7859:
--
[~wangwj], thank you for the debugging, patch and the suggestion. While your
patch probably works in the above mentioned scenario, it just works around the
problem. I think the issue is that this code below states that it compares the
demands but it instead says, if {{s1>0 ? "s1 is bigger": (s2>0 ? "s2 is bigger"
: "=")}}.
{code:java}
private int compareDemand(Schedulable s1, Schedulable s2) {
int res = 0;
Resource demand1 = s1.getDemand();
Resource demand2 = s2.getDemand();
if (demand1.equals(Resources.none()) && Resources.greaterThan(
RESOURCE_CALCULATOR, null, demand2, Resources.none())) {
res = 1;
} else if (demand2.equals(Resources.none()) && Resources.greaterThan(
RESOURCE_CALCULATOR, null, demand1, Resources.none())) {
res = -1;
}
return res;
}
{code}
This is not what it should do. This is not comparison and I think this is
causing the starvation of your second queue picking the first one if it has any
demand. Could you try removing your workaround and fixing the {{compare}}
function above? I think that will solve the issue with much less lines of code.
> New feature: add queue scheduling deadLine in fairScheduler.
>
>
> Key: YARN-7859
> URL: https://issues.apache.org/jira/browse/YARN-7859
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: wangwj
>Assignee: wangwj
>Priority: Major
> Labels: fairscheduler, features, patch
> Fix For: 3.0.0
>
> Attachments: YARN-7859-v1.patch, log, screenshot-1.png,
> screenshot-3.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> As everyone knows.In FairScheduler the phenomenon of queue scheduling
> starvation often occurs when the number of cluster jobs is large.The App in
> one or more queue are pending.So I have thought a way to solve this
> problem.Add queue scheduling deadLine in fairScheduler.When a queue is not
> scheduled for FairScheduler within a specified time.We mandatory scheduler it!
> On the basis of the above, I propose this issue...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7890) NPE during container relaunch
[
https://issues.apache.org/jira/browse/YARN-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353085#comment-16353085
]
genericqa commented on YARN-7890:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 1 unchanged - 1 fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
42s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7890 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909297/YARN-7890.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux a92da789249c 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33e6cdb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19606/testReport/ |
| Max. process+thread count | 397 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19606/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Updated] (YARN-7516) Security check for trusted docker image
[ https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7516: Attachment: YARN-7516.017.patch > Security check for trusted docker image > --- > > Key: YARN-7516 > URL: https://issues.apache.org/jira/browse/YARN-7516 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-7516.001.patch, YARN-7516.002.patch, > YARN-7516.003.patch, YARN-7516.004.patch, YARN-7516.005.patch, > YARN-7516.006.patch, YARN-7516.007.patch, YARN-7516.008.patch, > YARN-7516.009.patch, YARN-7516.010.patch, YARN-7516.011.patch, > YARN-7516.012.patch, YARN-7516.013.patch, YARN-7516.014.patch, > YARN-7516.015.patch, YARN-7516.016.patch, YARN-7516.017.patch > > > Hadoop YARN Services can support using private docker registry image or > docker image from docker hub. In current implementation, Hadoop security is > enforced through username and group membership, and enforce uid:gid > consistency in docker container and distributed file system. There is cloud > use case for having ability to run untrusted docker image on the same cluster > for testing. > The basic requirement for untrusted container is to ensure all kernel and > root privileges are dropped, and there is no interaction with distributed > file system to avoid contamination. We can probably enforce detection of > untrusted docker image by checking the following: > # If docker image is from public docker hub repository, the container is > automatically flagged as insecure, and disk volume mount are disabled > automatically, and drop all kernel capabilities. > # If docker image is from private repository in docker hub, and there is a > white list to allow the private repository, disk volume mount is allowed, > kernel capabilities follows the allowed list. > # If docker image is from private trusted registry with image name like > "private.registry.local:5000/centos", and white list allows this private > trusted repository. Disk volume mount is allowed, kernel capabilities > follows the allowed list. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7516) Security check for trusted docker image
[ https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353102#comment-16353102 ] Eric Yang commented on YARN-7516: - - Added more test cases in patch 17. > Security check for trusted docker image > --- > > Key: YARN-7516 > URL: https://issues.apache.org/jira/browse/YARN-7516 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-7516.001.patch, YARN-7516.002.patch, > YARN-7516.003.patch, YARN-7516.004.patch, YARN-7516.005.patch, > YARN-7516.006.patch, YARN-7516.007.patch, YARN-7516.008.patch, > YARN-7516.009.patch, YARN-7516.010.patch, YARN-7516.011.patch, > YARN-7516.012.patch, YARN-7516.013.patch, YARN-7516.014.patch, > YARN-7516.015.patch, YARN-7516.016.patch, YARN-7516.017.patch > > > Hadoop YARN Services can support using private docker registry image or > docker image from docker hub. In current implementation, Hadoop security is > enforced through username and group membership, and enforce uid:gid > consistency in docker container and distributed file system. There is cloud > use case for having ability to run untrusted docker image on the same cluster > for testing. > The basic requirement for untrusted container is to ensure all kernel and > root privileges are dropped, and there is no interaction with distributed > file system to avoid contamination. We can probably enforce detection of > untrusted docker image by checking the following: > # If docker image is from public docker hub repository, the container is > automatically flagged as insecure, and disk volume mount are disabled > automatically, and drop all kernel capabilities. > # If docker image is from private repository in docker hub, and there is a > white list to allow the private repository, disk volume mount is allowed, > kernel capabilities follows the allowed list. > # If docker image is from private trusted registry with image name like > "private.registry.local:5000/centos", and white list allows this private > trusted repository. Disk volume mount is allowed, kernel capabilities > follows the allowed list. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3895) Support ACLs in ATSv2
[ https://issues.apache.org/jira/browse/YARN-3895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353115#comment-16353115 ] Vrushali C commented on YARN-3895: -- Here is the design after several rounds of discussions in the community. Thanks [~jlowe] , [~jrottinghuis] [~lohit] for discussing with us (me, [~rohithsharma] and [~varun_saxena]). - We will go with the domain concept as in ATSv1. Entities will be written with a TimelineDomain (like in ATSv1) and there will be putDomain calls just like ATSv1. - The domain information will be persisted to the backend in a domain table. - The domain information will also be retained in the TimelineCollector. This now makes the Timeline Collector stateful. - If a timeline collector goes down (for whatever reason) and comes back up, it knows which app ids it had in memory. The collector will in this specific case, “refresh” it’s ACLs state by reading back from HBase, the domain ids for those app ids. - Each time an entity is received by the collector, it looks up the app id + domain id in it’s memory and appends the TimelineDomain to entity. - The entity when written to HBase has not only the domain id but also the Timeline Domain information. - Thus, each row in HBase will have the ACLs info which can be used for filtering at read time. - When a read request comes in, the user and user’s group will be sent to the HBase cluster in the scan/get request and a check will be performed on the region server to determine if this user is allowed to read that entity or not based on the user & group membership. - Since we want to evaluate group of group memberships, this check will be a UserGroupInformation check just like it’s done in any other yarn ACL evaluation. This implies, the yarn cluster AND the HBase cluster have to have the same username & group ldap mappings so that evaluation checks will work as expected. - I believe this would be done within a coprocessor but I will check if there is any other way to run java code as part of scan column value filter operation. - If the querying user is an yarn admin, then no checks are necessary. - In case the ACLs for a domain ids need to be updated, that will mean scanning through the set of entities for that application id and updating the domain information for those. - The domain table will have domain id as row key and other fields in the TimelineDomain object as columns. Perhaps only one column family is fine. Details per table in HBase: - Domain table schema Rowkey : domain id ColumnFamily: i (stands for info) Columns: (listing a few here, there can be others) - application_id - created time - description - modified time - owner - readers - writers (not used but can be stored for completeness) We can consider setting compression for this table at a high level, since we do not anticipate reading frequently from this table. - Entity table, SubApplication table, Application table. can store the domain id as a column and the fields in the domain object as separate columns. - FlowRun table. We can start with doing a union of ACLs for all applications within a flow run. - FlowActivity table. We can start by doing a union of ACLs for all runs in a flow in that time frame. This may turn out to a bit more involved. Let’s discuss on the jira we file for this. thanks Vrushali > Support ACLs in ATSv2 > - > > Key: YARN-3895 > URL: https://issues.apache.org/jira/browse/YARN-3895 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Vrushali C >Priority: Major > Labels: YARN-5355 > > This JIRA is to keep track of authorization support design discussions for > both readers and collectors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7516) Security check for trusted docker image
[ https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353125#comment-16353125 ] Eric Yang commented on YARN-7516: - Summary of the security check is reflected in the following table: || ||Privileged Enabled|| ||Privileged Disabled|| || |Registry|Trusted|Untrusted|Trusted|Untrusted| |Capability|ADD|DROP ALL|ADD allowed|DROP ALL| |Mount|RW, Read-only|Disable|RW, Read-only|Disable| |Device|ADD|DROP|ADD|DROP| |Launch Command|ADD|DROP|ADD|DROP| The current implementation is mostly symmetric with exception that privileged enabled allows full root access to host devices, where privileged disabled would limit to allowed list only. > Security check for trusted docker image > --- > > Key: YARN-7516 > URL: https://issues.apache.org/jira/browse/YARN-7516 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-7516.001.patch, YARN-7516.002.patch, > YARN-7516.003.patch, YARN-7516.004.patch, YARN-7516.005.patch, > YARN-7516.006.patch, YARN-7516.007.patch, YARN-7516.008.patch, > YARN-7516.009.patch, YARN-7516.010.patch, YARN-7516.011.patch, > YARN-7516.012.patch, YARN-7516.013.patch, YARN-7516.014.patch, > YARN-7516.015.patch, YARN-7516.016.patch, YARN-7516.017.patch > > > Hadoop YARN Services can support using private docker registry image or > docker image from docker hub. In current implementation, Hadoop security is > enforced through username and group membership, and enforce uid:gid > consistency in docker container and distributed file system. There is cloud > use case for having ability to run untrusted docker image on the same cluster > for testing. > The basic requirement for untrusted container is to ensure all kernel and > root privileges are dropped, and there is no interaction with distributed > file system to avoid contamination. We can probably enforce detection of > untrusted docker image by checking the following: > # If docker image is from public docker hub repository, the container is > automatically flagged as insecure, and disk volume mount are disabled > automatically, and drop all kernel capabilities. > # If docker image is from private repository in docker hub, and there is a > white list to allow the private repository, disk volume mount is allowed, > kernel capabilities follows the allowed list. > # If docker image is from private trusted registry with image name like > "private.registry.local:5000/centos", and white list allows this private > trusted repository. Disk volume mount is allowed, kernel capabilities > follows the allowed list. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7849) TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error
[
https://issues.apache.org/jira/browse/YARN-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353127#comment-16353127
]
Hudson commented on YARN-7849:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13614 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13614/])
YARN-7849. TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization (jlowe:
rev 60656bcecadd80e28c81bc943b44abf13d20abae)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYarnClusterNodeUtilization.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
> TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to
> heartbeat sync error
> --
>
> Key: YARN-7849
> URL: https://issues.apache.org/jira/browse/YARN-7849
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.1.0, 2.9.1, 3.0.1, 2.8.4
>Reporter: Jason Lowe
>Assignee: Botong Huang
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1, 2.8.4
>
> Attachments: YARN-7849-branch-2.8.v1.patch, YARN-7849.v1.patch,
> YARN-7849.v2.patch, YARN-7849.v3.patch
>
>
> testUpdateNodeUtilization is failing. From a branch-2.8 run:
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 13.013 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 12.961 sec <<< FAILURE!
> java.lang.AssertionError: Containers Utillization not propagated to RMNode
> expected:<> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:227)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7849) TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error
[
https://issues.apache.org/jira/browse/YARN-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353129#comment-16353129
]
Botong Huang commented on YARN-7849:
Sounds good, thanks [~jlowe]!
> TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to
> heartbeat sync error
> --
>
> Key: YARN-7849
> URL: https://issues.apache.org/jira/browse/YARN-7849
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.1.0, 2.9.1, 3.0.1, 2.8.4
>Reporter: Jason Lowe
>Assignee: Botong Huang
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1, 2.8.4
>
> Attachments: YARN-7849-branch-2.8.v1.patch, YARN-7849.v1.patch,
> YARN-7849.v2.patch, YARN-7849.v3.patch
>
>
> testUpdateNodeUtilization is failing. From a branch-2.8 run:
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 13.013 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 12.961 sec <<< FAILURE!
> java.lang.AssertionError: Containers Utillization not propagated to RMNode
> expected:<> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:227)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7516) Security check for trusted docker image
[
https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353137#comment-16353137
]
genericqa commented on YARN-7516:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-7516 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7516 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909316/YARN-7516.017.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19608/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Security check for trusted docker image
> ---
>
> Key: YARN-7516
> URL: https://issues.apache.org/jira/browse/YARN-7516
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-7516.001.patch, YARN-7516.002.patch,
> YARN-7516.003.patch, YARN-7516.004.patch, YARN-7516.005.patch,
> YARN-7516.006.patch, YARN-7516.007.patch, YARN-7516.008.patch,
> YARN-7516.009.patch, YARN-7516.010.patch, YARN-7516.011.patch,
> YARN-7516.012.patch, YARN-7516.013.patch, YARN-7516.014.patch,
> YARN-7516.015.patch, YARN-7516.016.patch, YARN-7516.017.patch
>
>
> Hadoop YARN Services can support using private docker registry image or
> docker image from docker hub. In current implementation, Hadoop security is
> enforced through username and group membership, and enforce uid:gid
> consistency in docker container and distributed file system. There is cloud
> use case for having ability to run untrusted docker image on the same cluster
> for testing.
> The basic requirement for untrusted container is to ensure all kernel and
> root privileges are dropped, and there is no interaction with distributed
> file system to avoid contamination. We can probably enforce detection of
> untrusted docker image by checking the following:
> # If docker image is from public docker hub repository, the container is
> automatically flagged as insecure, and disk volume mount are disabled
> automatically, and drop all kernel capabilities.
> # If docker image is from private repository in docker hub, and there is a
> white list to allow the private repository, disk volume mount is allowed,
> kernel capabilities follows the allowed list.
> # If docker image is from private trusted registry with image name like
> "private.registry.local:5000/centos", and white list allows this private
> trusted repository. Disk volume mount is allowed, kernel capabilities
> follows the allowed list.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5015) Unify restart policies across AM and container restarts
[ https://issues.apache.org/jira/browse/YARN-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353142#comment-16353142 ] Chandni Singh commented on YARN-5015: - Hi [~hex108] Are you still working on this? We have a use case for Yarn Service to support validity interval for container relaunch. > Unify restart policies across AM and container restarts > --- > > Key: YARN-5015 > URL: https://issues.apache.org/jira/browse/YARN-5015 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Jun Gong >Priority: Major > Labels: oct16-medium > Attachments: YARN-5015.01.patch > > > We support AM restart and container restarts - however the two have slightly > different capabilities. We should unify them. There's no reason for them to > be different. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5015) Unify restart policies across AM and container restarts
[
https://issues.apache.org/jira/browse/YARN-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353149#comment-16353149
]
genericqa commented on YARN-5015:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-5015 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5015 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12802918/YARN-5015.01.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19609/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Unify restart policies across AM and container restarts
> ---
>
> Key: YARN-5015
> URL: https://issues.apache.org/jira/browse/YARN-5015
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Jun Gong
>Priority: Major
> Labels: oct16-medium
> Attachments: YARN-5015.01.patch
>
>
> We support AM restart and container restarts - however the two have slightly
> different capabilities. We should unify them. There's no reason for them to
> be different.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7657) Queue Mapping could provide options to provide 'user' specific auto-created queues under a specified group parent queue
[
https://issues.apache.org/jira/browse/YARN-7657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353147#comment-16353147
]
genericqa commented on YARN-7657:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 19 new + 110 unchanged - 0 fixed = 129 total (was 110)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 38s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoCreatedQueuePreemption
|
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7657 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909295/YARN-7657.1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 8975c5d07129 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33e6cdb |
| maven | version: Apache Maven 3.3.9 |
| Default
[jira] [Commented] (YARN-7889) Missing kerberos token when check for RM REST API availability
[ https://issues.apache.org/jira/browse/YARN-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353154#comment-16353154 ] Jian He commented on YARN-7889: --- Patch, lgtm, One side issue noticed is that, till now, the client library (YarnClient, AMRMClient) YARN exposes have builtin retry, and applications may assume that the retry is handled automatically. But the ApiServiceClient doesn't have retry built-in, this may be a caveat for applications later. I think we can deal with this later when requests come. > Missing kerberos token when check for RM REST API availability > -- > > Key: YARN-7889 > URL: https://issues.apache.org/jira/browse/YARN-7889 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-7889.001.patch, YARN-7889.002.patch > > > When checking for which resource manager can be used for REST API request, > client side must send kerberos token to REST API end point. The checking > mechanism is currently missing the kerberos token. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7889) Missing kerberos token when check for RM REST API availability
[
https://issues.apache.org/jira/browse/YARN-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353179#comment-16353179
]
genericqa commented on YARN-7889:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 9s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api:
The patch generated 2 new + 3 unchanged - 1 fixed = 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7889 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909307/YARN-7889.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2b599ab0682a 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 60656bc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19607/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services-api.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit
[jira] [Updated] (YARN-7889) Missing kerberos token when check for RM REST API availability
[ https://issues.apache.org/jira/browse/YARN-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7889: Attachment: YARN-7889.003.patch > Missing kerberos token when check for RM REST API availability > -- > > Key: YARN-7889 > URL: https://issues.apache.org/jira/browse/YARN-7889 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-7889.001.patch, YARN-7889.002.patch, > YARN-7889.003.patch > > > When checking for which resource manager can be used for REST API request, > client side must send kerberos token to REST API end point. The checking > mechanism is currently missing the kerberos token. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7789) Should fail RM if 3rd resource type is configured but RM uses DefaultResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zian Chen updated YARN-7789: Attachment: YARN-7789.001.patch > Should fail RM if 3rd resource type is configured but RM uses > DefaultResourceCalculator > --- > > Key: YARN-7789 > URL: https://issues.apache.org/jira/browse/YARN-7789 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sumana Sathish >Assignee: Zian Chen >Priority: Critical > Attachments: YARN-7789.001.patch > > > We may need to revisit this behavior: Currently, RM doesn't fail if 3rd > resource type is configured, allocated containers will be automatically > assigned minimum allocation for all resource types except memory, this makes > really hard for troubleshooting. I prefer to fail RM if 3rd or more resource > type is configured inside resource-types.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7889) Missing kerberos token when check for RM REST API availability
[ https://issues.apache.org/jira/browse/YARN-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7889: Attachment: (was: YARN-7889.003.patch) > Missing kerberos token when check for RM REST API availability > -- > > Key: YARN-7889 > URL: https://issues.apache.org/jira/browse/YARN-7889 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-7889.001.patch, YARN-7889.002.patch, > YARN-7889.003.patch > > > When checking for which resource manager can be used for REST API request, > client side must send kerberos token to REST API end point. The checking > mechanism is currently missing the kerberos token. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7889) Missing kerberos token when check for RM REST API availability
[ https://issues.apache.org/jira/browse/YARN-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7889: Attachment: YARN-7889.003.patch > Missing kerberos token when check for RM REST API availability > -- > > Key: YARN-7889 > URL: https://issues.apache.org/jira/browse/YARN-7889 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-7889.001.patch, YARN-7889.002.patch, > YARN-7889.003.patch > > > When checking for which resource manager can be used for REST API request, > client side must send kerberos token to REST API end point. The checking > mechanism is currently missing the kerberos token. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7789) Should fail RM if 3rd resource type is configured but RM uses DefaultResourceCalculator
[
https://issues.apache.org/jira/browse/YARN-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353193#comment-16353193
]
Suma Shivaprasad commented on YARN-7789:
Patch LGTM. A few minor comments.
1.Can we reword the failure message from
+ throw new YarnRuntimeException("Invalid resource types specified when" + + "
enabling GPU, resource manager should fail if 3rd resource" + + " type is
configured but RM uses DefaultResourceCalculator");
to convey that Default
+ throw new YarnRuntimeException("RM uses DefaultResourceCalculator which used
only memory as resource-type but invalid resource-types specified " + ) . Use DomainantResourceCalculator instead to make effective
use of these resource-types "
2. Please add a UT for the same.
> Should fail RM if 3rd resource type is configured but RM uses
> DefaultResourceCalculator
> ---
>
> Key: YARN-7789
> URL: https://issues.apache.org/jira/browse/YARN-7789
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Sumana Sathish
>Assignee: Zian Chen
>Priority: Critical
> Attachments: YARN-7789.001.patch
>
>
> We may need to revisit this behavior: Currently, RM doesn't fail if 3rd
> resource type is configured, allocated containers will be automatically
> assigned minimum allocation for all resource types except memory, this makes
> really hard for troubleshooting. I prefer to fail RM if 3rd or more resource
> type is configured inside resource-types.xml.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7889) Missing kerberos token when check for RM REST API availability
[
https://issues.apache.org/jira/browse/YARN-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353213#comment-16353213
]
genericqa commented on YARN-7889:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 7s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api:
The patch generated 1 new + 3 unchanged - 1 fixed = 4 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 44s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7889 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909339/YARN-7889.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 49fca9f36514 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 60656bc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19610/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services-api.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19610/testReport/ |
| Max. process+thread count | 409 (vs. ulimit of 5
[jira] [Created] (YARN-7897) Invalid NM log & NM UI link published on Yarn UI when container fails
Yesha Vora created YARN-7897: Summary: Invalid NM log & NM UI link published on Yarn UI when container fails Key: YARN-7897 URL: https://issues.apache.org/jira/browse/YARN-7897 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Yesha Vora Steps: 1) Launch Httpd example via rest api in unsecure mode 2) container_e04_1517875972784_0001_01_02 fails with "Unable to find image 'centos/httpd-24-centos7:latest" 3) Go To RM UI2 to debug issue. The Yarn app attempt page has incorrect Value for Logs and Nodemanager UI Logs = N/A Nodemanager UI = http://nmhost:0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7897) Invalid NM log & NM UI link published on Yarn UI when container fails
[ https://issues.apache.org/jira/browse/YARN-7897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora updated YARN-7897: - Attachment: Screen Shot 2018-02-05 at 4.52.59 PM.png > Invalid NM log & NM UI link published on Yarn UI when container fails > - > > Key: YARN-7897 > URL: https://issues.apache.org/jira/browse/YARN-7897 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Yesha Vora >Priority: Major > Attachments: Screen Shot 2018-02-05 at 4.52.59 PM.png > > > Steps: > 1) Launch Httpd example via rest api in unsecure mode > 2) container_e04_1517875972784_0001_01_02 fails with "Unable to find > image 'centos/httpd-24-centos7:latest" > 3) Go To RM UI2 to debug issue. > The Yarn app attempt page has incorrect Value for Logs and Nodemanager UI > Logs = N/A > Nodemanager UI = http://nmhost:0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7890) NPE during container relaunch
[
https://issues.apache.org/jira/browse/YARN-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353223#comment-16353223
]
Billie Rinaldi commented on YARN-7890:
--
+1 for patch 002. This fixes the NPE. Thanks for the patch, [~jlowe], and
thanks for the review, [~shaneku...@gmail.com]! I'll commit this one.
> NPE during container relaunch
> -
>
> Key: YARN-7890
> URL: https://issues.apache.org/jira/browse/YARN-7890
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Billie Rinaldi
>Assignee: Jason Lowe
>Priority: Major
> Attachments: YARN-7890.001.patch, YARN-7890.002.patch
>
>
> While running a recent build of trunk, I saw the following:
> {noformat}
> 2018-02-02 21:02:40,026 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_e02_1517604848419_0002_01_04 transitioned from
> RELAUNCHING to RUNNING
> 2018-02-02 21:02:40,026 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch:
> Failed to relaunch container.
> java.lang.NullPointerException
> at
> java.util.Collections$UnmodifiableCollection.(Collections.java:1026)
> at
> java.util.Collections$UnmodifiableList.(Collections.java:1302)
> at java.util.Collections.unmodifiableList(Collections.java:1287)
> at
> org.apache.hadoop.yarn.server.nodemanager.executor.ContainerStartContext.getFilecacheDirs(ContainerStartContext.java:200)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.buildContainerRuntimeContext(LinuxContainerExecutor.java:651)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:546)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:465)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:49)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7890) NPE during container relaunch
[
https://issues.apache.org/jira/browse/YARN-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353243#comment-16353243
]
Hudson commented on YARN-7890:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13615 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13615/])
YARN-7890. NPE during container relaunch. Contributed by Jason Lowe (billie:
rev a196ee9362a1b35e5de20ee519f7c544ab1588e1)
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerRelaunch.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerRelaunch.java
> NPE during container relaunch
> -
>
> Key: YARN-7890
> URL: https://issues.apache.org/jira/browse/YARN-7890
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Billie Rinaldi
>Assignee: Jason Lowe
>Priority: Major
> Attachments: YARN-7890.001.patch, YARN-7890.002.patch
>
>
> While running a recent build of trunk, I saw the following:
> {noformat}
> 2018-02-02 21:02:40,026 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_e02_1517604848419_0002_01_04 transitioned from
> RELAUNCHING to RUNNING
> 2018-02-02 21:02:40,026 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch:
> Failed to relaunch container.
> java.lang.NullPointerException
> at
> java.util.Collections$UnmodifiableCollection.(Collections.java:1026)
> at
> java.util.Collections$UnmodifiableList.(Collections.java:1302)
> at java.util.Collections.unmodifiableList(Collections.java:1287)
> at
> org.apache.hadoop.yarn.server.nodemanager.executor.ContainerStartContext.getFilecacheDirs(ContainerStartContext.java:200)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.buildContainerRuntimeContext(LinuxContainerExecutor.java:651)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:546)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:465)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerRelaunch.call(ContainerRelaunch.java:49)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7739) DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation
[ https://issues.apache.org/jira/browse/YARN-7739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353251#comment-16353251 ] Wangda Tan commented on YARN-7739: -- [~templedf] , could you help to check this patch? > DefaultAMSProcessor should properly check customized resource types against > minimum/maximum allocation > -- > > Key: YARN-7739 > URL: https://issues.apache.org/jira/browse/YARN-7739 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7339.002.patch, YARN-7739.001.patch > > > Currently, YARN RM reject requested resource if memory or vcores are less > than 0 or greater than maximum allocation. We should run the check for > customized resource types as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5742) Serve aggregated logs of historical apps from timeline service
[ https://issues.apache.org/jira/browse/YARN-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353254#comment-16353254 ] Wangda Tan commented on YARN-5742: -- [~rohithsharma] , do you still plan to work on this item? Is it still a blocker/critical since we have workarounds. > Serve aggregated logs of historical apps from timeline service > -- > > Key: YARN-5742 > URL: https://issues.apache.org/jira/browse/YARN-5742 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Rohith Sharma K S >Priority: Blocker > Attachments: YARN-5742-POC-v0.patch > > > ATSv1.5 daemon has servlet to serve aggregated logs. But enabling only ATSv2, > does not serve logs from CLI and UI for completed application. Log serving > story has completely broken in ATSv2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7292) Revisit Resource Profile Behavior
[ https://issues.apache.org/jira/browse/YARN-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353255#comment-16353255 ] Sunil G commented on YARN-7292: --- It seems Findbugs warning which is shown in the report and the link are different. I ll re-run one more time. > Revisit Resource Profile Behavior > - > > Key: YARN-7292 > URL: https://issues.apache.org/jira/browse/YARN-7292 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7292.002.patch, YARN-7292.003.patch, > YARN-7292.wip.001.patch > > > Had discussions with [~templedf], [~vvasudev], [~sunilg] offline. There're a > couple of resource profile related behaviors might need to be updated: > 1) Configure resource profile in server side or client side: > Currently resource profile can be only configured centrally: > - Advantages: > A given resource profile has a the same meaning in the cluster. It won’t > change when we run different apps in different configurations. A job can run > under Amazon’s G2.8X can also run on YARN with G2.8X profile. A side benefit > is YARN scheduler can potentially do better bin packing. > - Disadvantages: > Hard for applications to add their own resource profiles. > 2) Do we really need mandatory resource profiles such as > minimum/maximum/default? > 3) Should we send resource profile name inside ResourceRequest, or should > client/AM translate it to resource and set it to the existing resource > fields? > 4) Related to above, should we allow resource overrides or client/AM should > send final resource to RM? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5848) public/crossdomain.xml is problematic
[ https://issues.apache.org/jira/browse/YARN-5848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353260#comment-16353260 ] Wangda Tan commented on YARN-5848: -- Thanks [~aw] for reporting this issue. [~sunilg] , could you help to check this issue? I saw two crossdomain.xml under ui2 project. > public/crossdomain.xml is problematic > - > > Key: YARN-5848 > URL: https://issues.apache.org/jira/browse/YARN-5848 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.0.0-alpha2, 3.1.0 >Reporter: Allen Wittenauer >Priority: Blocker > > crossdomain.xml should really have an ASF header in it and be in the src > directory somewhere. There's zero reason for it to have RAT exception given > that comments are possible in xml files. It's also not in a standard maven > location, which should really be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5848) public/crossdomain.xml is problematic
[ https://issues.apache.org/jira/browse/YARN-5848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353267#comment-16353267 ] Sunil G commented on YARN-5848: --- Sure. I will check and update. > public/crossdomain.xml is problematic > - > > Key: YARN-5848 > URL: https://issues.apache.org/jira/browse/YARN-5848 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.0.0-alpha2, 3.1.0 >Reporter: Allen Wittenauer >Priority: Blocker > > crossdomain.xml should really have an ASF header in it and be in the src > directory somewhere. There's zero reason for it to have RAT exception given > that comments are possible in xml files. It's also not in a standard maven > location, which should really be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7655) avoid AM preemption caused by RRs for specific nodes or racks
[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353279#comment-16353279
]
Steven Rand commented on YARN-7655:
---
The concern I have with all three RRs being the same size is that we don't
necessarily consider the {{NODE_LOCAL}} RR for preemption. My understanding is
that we might wind up preempting for one of the other RRs, in which case we're
no longer testing the change to the production code. Let me know if I'm
misunderstanding though.
> avoid AM preemption caused by RRs for specific nodes or racks
> -
>
> Key: YARN-7655
> URL: https://issues.apache.org/jira/browse/YARN-7655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Attachments: YARN-7655-001.patch, YARN-7655-002.patch
>
>
> We frequently see AM preemptions when
> {{starvedApp.getStarvedResourceRequests()}} in
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs
> that request containers on a specific node. Since this causes us to only
> consider one node to preempt containers on, the really good work that was
> done in YARN-5830 doesn't save us from AM preemption. Even though there might
> be multiple nodes on which we could preempt enough non-AM containers to
> satisfy the app's starvation, we often wind up preempting one or more AM
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM
> containers for an RR that specifies a node or rack, then we should instead
> expand the search space to consider all nodes. That way we take advantage of
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a
> patch with an initial implementation of this. We've been running it on a few
> clusters, and have seen AM preemptions drop from double-digit occurrences on
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is
> less likely to be allocated resources at the most specific locality level
> that it asked for. My opinion is that this tradeoff is worth it, but
> interested to hear what others think as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7789) Should fail RM if 3rd resource type is configured but RM uses DefaultResourceCalculator
[
https://issues.apache.org/jira/browse/YARN-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353290#comment-16353290
]
genericqa commented on YARN-7789:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 13s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7789 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909338/YARN-7789.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 7e51f8f9664d 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 60656bc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19611/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19611/testReport/ |
| Max. process+thread count | 839 (vs. ulimit of 5500) |
| modules
[jira] [Commented] (YARN-7866) [UI2] Kerberizing the UI doesn't give any warning or content when UI is accessed without kinit
[ https://issues.apache.org/jira/browse/YARN-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353302#comment-16353302 ] Wangda Tan commented on YARN-7866: -- Looks good, +1, thanks [~sunilg] > [UI2] Kerberizing the UI doesn't give any warning or content when UI is > accessed without kinit > -- > > Key: YARN-7866 > URL: https://issues.apache.org/jira/browse/YARN-7866 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Sumana Sathish >Assignee: Sunil G >Priority: Major > Attachments: Screen Shot 2018-02-03 at 3.22.13 PM.png, > YARN-7866.001.patch, YARN-7866.002.patch > > > Handle 401 error and show in UI > credit to [~ssath...@hortonworks.com] for finding this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7655) avoid AM preemption caused by RRs for specific nodes or racks
[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Rand updated YARN-7655:
--
Attachment: YARN-7655-003.patch
> avoid AM preemption caused by RRs for specific nodes or racks
> -
>
> Key: YARN-7655
> URL: https://issues.apache.org/jira/browse/YARN-7655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Attachments: YARN-7655-001.patch, YARN-7655-002.patch,
> YARN-7655-003.patch
>
>
> We frequently see AM preemptions when
> {{starvedApp.getStarvedResourceRequests()}} in
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs
> that request containers on a specific node. Since this causes us to only
> consider one node to preempt containers on, the really good work that was
> done in YARN-5830 doesn't save us from AM preemption. Even though there might
> be multiple nodes on which we could preempt enough non-AM containers to
> satisfy the app's starvation, we often wind up preempting one or more AM
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM
> containers for an RR that specifies a node or rack, then we should instead
> expand the search space to consider all nodes. That way we take advantage of
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a
> patch with an initial implementation of this. We've been running it on a few
> clusters, and have seen AM preemptions drop from double-digit occurrences on
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is
> less likely to be allocated resources at the most specific locality level
> that it asked for. My opinion is that this tradeoff is worth it, but
> interested to hear what others think as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7838) Support AND/OR constraints in Distributed Shell
[
https://issues.apache.org/jira/browse/YARN-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YARN-7838:
--
Attachment: YARN-7838.001.patch
> Support AND/OR constraints in Distributed Shell
> ---
>
> Key: YARN-7838
> URL: https://issues.apache.org/jira/browse/YARN-7838
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-7838.001.patch, YARN-7838.prelim.patch
>
>
> Extending DS placement spec syntax to support AND/OR constraints, something
> like
> {code}
> // simple
> -placement_spec foo=4,AND(NOTIN,NODE,foo:NOTIN,NODE,bar)
> // nested
> -placement_spec foo=4,AND(NOTIN,NODE,foo:OR(IN,NODE,moo:IN,NODE,bar))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7838) Support AND/OR constraints in Distributed Shell
[
https://issues.apache.org/jira/browse/YARN-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353324#comment-16353324
]
Weiwei Yang commented on YARN-7838:
---
Hi [~asuresh]
I just submitted a formal v1 patch, this patch adds a
{{PlacementConstraintParser}}, and {{PlacementSpec}} now simply calls
{{parsePlacementSpec}} to parse a string expression. Syntax is just like before
{code}
Tag1=N1,P1:Tag2=N2,P2:...:TagN=Nn,Pn
{code}
where each KV pair for tag represents source tag and number of allocations,
such as {{foo=3}}.
where each {{Pn}} can be any form of a valid constraint expression (include
nested forms), such as
{code}
in,node,foo,bar
notin,node,foo,bar,1,2
and(notin,node,foo:notin,node,bar)
and(notin,node,foo:or(in,node.bar:in,node,moo))
{code}
a comprehensive UT can be found in {{TestPlacementConstraintParser}}. I have
also tried end-to-end testing with different combination of constraints,
includes
{code}
-placement_spec foo=4,notin,node,foo
-placement_spec foo=3,NOTIN,NODE,foo:bar=1,IN,NODE,foo
-placement_spec "foo=4,AND(notin,node,foo:cardinality,node,foo,0,2)"
{code}
all seem to work fine to me. Please check and let me know your feedback.
Thanks
> Support AND/OR constraints in Distributed Shell
> ---
>
> Key: YARN-7838
> URL: https://issues.apache.org/jira/browse/YARN-7838
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-7838.001.patch, YARN-7838.prelim.patch
>
>
> Extending DS placement spec syntax to support AND/OR constraints, something
> like
> {code}
> // simple
> -placement_spec foo=4,AND(NOTIN,NODE,foo:NOTIN,NODE,bar)
> // nested
> -placement_spec foo=4,AND(NOTIN,NODE,foo:OR(IN,NODE,moo:IN,NODE,bar))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7572) Make the service status output more readable
[ https://issues.apache.org/jira/browse/YARN-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353326#comment-16353326 ] Eric Yang commented on YARN-7572: - [~csingh] I think we can also omit null value fields to clean up the output. > Make the service status output more readable > - > > Key: YARN-7572 > URL: https://issues.apache.org/jira/browse/YARN-7572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Chandni Singh >Priority: Major > Fix For: yarn-native-services > > > Currently the service status output is just a JSON spec, we can make it more > human readable -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7838) Support AND/OR constraints in Distributed Shell
[
https://issues.apache.org/jira/browse/YARN-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353362#comment-16353362
]
genericqa commented on YARN-7838:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
5s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 7 new + 141 unchanged - 0 fixed = 148 total (was 141) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 39s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
37s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
|
| | Found reliance on default encoding in
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.parsePlacementSpecs(String):in
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.parsePlacementSpecs(String):
new String(byte[]) At ApplicationMaster.java:[line 677] |
| | Found reliance on de
[jira] [Commented] (YARN-7655) avoid AM preemption caused by RRs for specific nodes or racks
[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353364#comment-16353364
]
genericqa commented on YARN-7655:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 40s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 57s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7655 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909357/YARN-7655-003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 399cbfc281bb 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a196ee9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19613/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19613/testReport/ |
| Max. process+thread count | 809 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/h
[jira] [Commented] (YARN-7292) Revisit Resource Profile Behavior
[
https://issues.apache.org/jira/browse/YARN-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353435#comment-16353435
]
genericqa commented on YARN-7292:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
10s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 8m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 0s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 298 unchanged - 11 fixed = 303 total (was 309) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 58s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
19s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
13s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 27s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 17s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
45s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF
[jira] [Commented] (YARN-7346) Fix compilation errors against hbase2 beta release
[ https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353449#comment-16353449 ] Rohith Sharma K S commented on YARN-7346: - It appears to be conditional compilation and ACLs story YARN-3895 are going to conflict each other. It would be nice if we can finish conditional compilation first as we discussed then we can move for ACLs without any modular issue. Otherwise I am seeing more complexity for conditional compilations or vice versa! cc :/ [~haibochen] [~vrushalic] > Fix compilation errors against hbase2 beta release > -- > > Key: YARN-7346 > URL: https://issues.apache.org/jira/browse/YARN-7346 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Vrushali C >Priority: Major > Attachments: YARN-7346.00.patch, YARN-7346.01.patch, > YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch > > > When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, > I got the following errors: > https://pastebin.com/Ms4jYEVB > This issue is to fix the compilation errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7889) Missing kerberos token when check for RM REST API availability
[ https://issues.apache.org/jira/browse/YARN-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353475#comment-16353475 ] Sunil G commented on YARN-7889: --- +1 from me as well. Looks fine. > Missing kerberos token when check for RM REST API availability > -- > > Key: YARN-7889 > URL: https://issues.apache.org/jira/browse/YARN-7889 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-7889.001.patch, YARN-7889.002.patch, > YARN-7889.003.patch > > > When checking for which resource manager can be used for REST API request, > client side must send kerberos token to REST API end point. The checking > mechanism is currently missing the kerberos token. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7866) [UI2] Kerberizing the UI doesn't give any warning or content when UI is accessed without kinit
[ https://issues.apache.org/jira/browse/YARN-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353478#comment-16353478 ] Sunil G commented on YARN-7866: --- Thanks [~leftnoteasy] I ll commit this shortly. > [UI2] Kerberizing the UI doesn't give any warning or content when UI is > accessed without kinit > -- > > Key: YARN-7866 > URL: https://issues.apache.org/jira/browse/YARN-7866 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Sumana Sathish >Assignee: Sunil G >Priority: Major > Attachments: Screen Shot 2018-02-03 at 3.22.13 PM.png, > YARN-7866.001.patch, YARN-7866.002.patch > > > Handle 401 error and show in UI > credit to [~ssath...@hortonworks.com] for finding this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7859) New feature: add queue scheduling deadLine in fairScheduler.
[ https://issues.apache.org/jira/browse/YARN-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353484#comment-16353484 ] wangwj commented on YARN-7859: -- Hi,[~yufeigu],thank you for your reply.:) 1、This phenomenon often happens in my cluster because of the large amount of jobs...I don't think this is a corner cases or particular issue..For example,one queue includes off-line batch processing job and real time stream processing job..Although this can be solved by disassembling the queue, but it restricts the flexibility of the queue. 2、If we add this new property to queue. Few code to use this property. This is different from maxAMShare. 3、If we take your suggestion to setting minShare to a smaller value for long-running job queue.Others queue will must be affected and one or more queues probably can't be shceduled for a long time. > New feature: add queue scheduling deadLine in fairScheduler. > > > Key: YARN-7859 > URL: https://issues.apache.org/jira/browse/YARN-7859 > Project: Hadoop YARN > Issue Type: New Feature > Components: fairscheduler >Affects Versions: 3.0.0 >Reporter: wangwj >Assignee: wangwj >Priority: Major > Labels: fairscheduler, features, patch > Fix For: 3.0.0 > > Attachments: YARN-7859-v1.patch, log, screenshot-1.png, > screenshot-3.png > > Original Estimate: 24h > Remaining Estimate: 24h > > As everyone knows.In FairScheduler the phenomenon of queue scheduling > starvation often occurs when the number of cluster jobs is large.The App in > one or more queue are pending.So I have thought a way to solve this > problem.Add queue scheduling deadLine in fairScheduler.When a queue is not > scheduled for FairScheduler within a specified time.We mandatory scheduler it! > On the basis of the above, I propose this issue... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
