[jira] [Assigned] (YARN-6118) Add javadoc for Resources.isNone
[ https://issues.apache.org/jira/browse/YARN-6118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G reassigned YARN-6118: - Assignee: Sunil G > Add javadoc for Resources.isNone > > > Key: YARN-6118 > URL: https://issues.apache.org/jira/browse/YARN-6118 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Sunil G >Priority: Minor > Labels: newbie > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6118) Add javadoc for Resources.isNone
[ https://issues.apache.org/jira/browse/YARN-6118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839772#comment-15839772 ] Sunil G commented on YARN-6118: --- I think you can assign yourself and provide a patch. > Add javadoc for Resources.isNone > > > Key: YARN-6118 > URL: https://issues.apache.org/jira/browse/YARN-6118 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Priority: Minor > Labels: newbie > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5641) Localizer leaves behind tarballs after container is complete
[ https://issues.apache.org/jira/browse/YARN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-5641: - Fix Version/s: (was: 2.9.0) My apologies, I missed the isAlive method calls on Process which are only available in JDK8. This has been reverted from branch-2. [~ebadger] please provide a separate patch for branch-2 that works with JDK7. > Localizer leaves behind tarballs after container is complete > > > Key: YARN-5641 > URL: https://issues.apache.org/jira/browse/YARN-5641 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5641.001.patch, YARN-5641.002.patch, > YARN-5641.003.patch, YARN-5641.004.patch, YARN-5641.005.patch, > YARN-5641.006.patch, YARN-5641.007.patch, YARN-5641.008.patch, > YARN-5641.009.patch, YARN-5641.009.patch, YARN-5641.010.patch > > > The localizer sometimes fails to clean up extracted tarballs leaving large > footprints that persist on the nodes indefinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6118) Add javadoc for Resources.isNone
[ https://issues.apache.org/jira/browse/YARN-6118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-6118: -- Assignee: (was: Sunil G) > Add javadoc for Resources.isNone > > > Key: YARN-6118 > URL: https://issues.apache.org/jira/browse/YARN-6118 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Priority: Minor > Labels: newbie > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3698) Make task attempt log files downloadable from all browsers
[
https://issues.apache.org/jira/browse/YARN-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreenath Somarajapuram updated YARN-3698:
-
Summary: Make task attempt log files downloadable from all browsers (was:
Correct make task attempt log files downloadable from all browsers)
> Make task attempt log files downloadable from all browsers
> --
>
> Key: YARN-3698
> URL: https://issues.apache.org/jira/browse/YARN-3698
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Sreenath Somarajapuram
>
> Please provide a variant of the link with the following headers set, this
> enables direct download of the file across all browsers. Without these we
> cannot provide a cross-browser download links.
> {code}
> Content-Disposition: attachment; filename="attempt-id.log"
> Content-Type of text/plain
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6123) [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues added or removed.
[
https://issues.apache.org/jira/browse/YARN-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839764#comment-15839764
]
Sunil G commented on YARN-6123:
---
1. {{PriorityUtilizationQueueOrderingPolicy.setQueues}} Do we need to make this
method under writelock to protect {{queues}} getting changed at runtime while
{{getAssignmentIterator}} is accessing the same (thought its a copy).
2. Few minor comments in test case:
a) {{bQ.getChildQueues().size()}} could we check this before calling
reinitialize once.
b) In {{checkEqualsToQueueSet}}, I think it may be better to check whether size
of {{queues}} and {{queueNames}} are equal and that can be a common assert in
side checkEqualsToQueueSet method. So we can avoid queue size assert in common
test case.
> [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated
> when childQueues added or removed.
> ---
>
> Key: YARN-6123
> URL: https://issues.apache.org/jira/browse/YARN-6123
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6123.001.patch
>
>
> YARN-5864 added queue ordering policy to ParentQueue, we need to make sure
> queues of QueueOrderingPolicy will be updated when any changes made for child
> queues.
> We need to add a test to make sure it works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6123) [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues added or removed.
[
https://issues.apache.org/jira/browse/YARN-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839764#comment-15839764
]
Sunil G edited comment on YARN-6123 at 1/26/17 2:40 PM:
Thanks [~wangda] for the patch. Few comments.
1. {{PriorityUtilizationQueueOrderingPolicy.setQueues}} Do we need to make this
method under writelock to protect {{queues}} getting changed at runtime while
{{getAssignmentIterator}} is accessing the same (thought its a copy).
2. Few minor comments in test case:
a) {{bQ.getChildQueues().size()}} could we check this before calling
reinitialize once.
b) In {{checkEqualsToQueueSet}}, I think it may be better to check whether size
of {{queues}} and {{queueNames}} are equal and that can be a common assert in
side checkEqualsToQueueSet method. So we can avoid queue size assert in common
test case.
was (Author: sunilg):
1. {{PriorityUtilizationQueueOrderingPolicy.setQueues}} Do we need to make this
method under writelock to protect {{queues}} getting changed at runtime while
{{getAssignmentIterator}} is accessing the same (thought its a copy).
2. Few minor comments in test case:
a) {{bQ.getChildQueues().size()}} could we check this before calling
reinitialize once.
b) In {{checkEqualsToQueueSet}}, I think it may be better to check whether size
of {{queues}} and {{queueNames}} are equal and that can be a common assert in
side checkEqualsToQueueSet method. So we can avoid queue size assert in common
test case.
> [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated
> when childQueues added or removed.
> ---
>
> Key: YARN-6123
> URL: https://issues.apache.org/jira/browse/YARN-6123
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6123.001.patch
>
>
> YARN-5864 added queue ordering policy to ParentQueue, we need to make sure
> queues of QueueOrderingPolicy will be updated when any changes made for child
> queues.
> We need to add a test to make sure it works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4658) Typo in o.a.h.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler comment
[
https://issues.apache.org/jira/browse/YARN-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839761#comment-15839761
]
Daniel Templeton commented on YARN-4658:
Could you add a unit test to cover the change?
Just kidding. +1 I'll commit later today.
> Typo in o.a.h.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
> comment
> --
>
> Key: YARN-4658
> URL: https://issues.apache.org/jira/browse/YARN-4658
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Daniel Templeton
>Assignee: Udai Kiran Potluri
> Attachments: YARN-4658.001.patch
>
>
> Comment in {{testContinuousSchedulingInterruptedException()}} is
> {code}
> // Add one nodes
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5641) Localizer leaves behind tarballs after container is complete
[ https://issues.apache.org/jira/browse/YARN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840153#comment-15840153 ] Xiao Chen commented on YARN-5641: - Thanks for the prompt response. I think the failure is understandable since the backport was clean. Thanks for working on the branch-2 patch. :) > Localizer leaves behind tarballs after container is complete > > > Key: YARN-5641 > URL: https://issues.apache.org/jira/browse/YARN-5641 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5641.001.patch, YARN-5641.002.patch, > YARN-5641.003.patch, YARN-5641.004.patch, YARN-5641.005.patch, > YARN-5641.006.patch, YARN-5641.007.patch, YARN-5641.008.patch, > YARN-5641.009.patch, YARN-5641.009.patch, YARN-5641.010-b2.patch, > YARN-5641.010.patch > > > The localizer sometimes fails to clean up extracted tarballs leaving large > footprints that persist on the nodes indefinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5641) Localizer leaves behind tarballs after container is complete
[ https://issues.apache.org/jira/browse/YARN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-5641: -- Attachment: YARN-5641.010-b2.patch Apologies for the bad Java 7 patch! I'll be sure to compile against Java 7 for anything that's targeted to anything other than trunk from now on. Here's a new patch for branch-2 that fixes the issues. > Localizer leaves behind tarballs after container is complete > > > Key: YARN-5641 > URL: https://issues.apache.org/jira/browse/YARN-5641 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5641.001.patch, YARN-5641.002.patch, > YARN-5641.003.patch, YARN-5641.004.patch, YARN-5641.005.patch, > YARN-5641.006.patch, YARN-5641.007.patch, YARN-5641.008.patch, > YARN-5641.009.patch, YARN-5641.009.patch, YARN-5641.010-b2.patch, > YARN-5641.010.patch > > > The localizer sometimes fails to clean up extracted tarballs leaving large > footprints that persist on the nodes indefinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6124) Make CapacityScheduler Preemption Config can be enabled / disabled / updated without restarting RM
[ https://issues.apache.org/jira/browse/YARN-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6124: - Attachment: YARN-6124.wip.1.patch Attached WIP patch to demonstrate what I plan to do: We will add a PCPP to schedulingMonitor no matter under any case. And it will periodically check if any reinitialize happens on CS side, if reinitialize happens and preemption enabled, it will sync latest preemption configs and refresh its local state. > Make CapacityScheduler Preemption Config can be enabled / disabled / updated > without restarting RM > -- > > Key: YARN-6124 > URL: https://issues.apache.org/jira/browse/YARN-6124 > Project: Hadoop YARN > Issue Type: Task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6124.wip.1.patch > > > Now enabled / disable / update CapacityScheduler preemption config requires > restart RM. This is inconvenient when admin wants to make changes to > preemption config. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-6013) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[
https://issues.apache.org/jira/browse/YARN-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Rand updated YARN-6013:
--
Comment: was deleted
(was: A bit more information: I've isolated the problem to the private class
{{Connection}} within {{Client}}, specifically the method {{sendRpcRequest}}.
More specifically, this block of code:
{code}
synchronized (sendRpcRequestLock) {
Future senderFuture = sendParamsExecutor.submit(new Runnable() {
@Override
public void run() {
try {
synchronized (ipcStreams.out) {
if (shouldCloseConnection.get()) {
return;
}
if (LOG.isDebugEnabled()) {
LOG.debug(getName() + " sending #" + call.id);
}
// RpcRequestHeader + RpcRequest
ipcStreams.sendRequest(buf.toByteArray());
ipcStreams.flush();
}
} catch (IOException e) {
// exception at this point would leave the connection in an
// unrecoverable state (eg half a call left on the wire).
// So, close the connection, killing any outstanding calls
markClosed(e);
} finally {
//the buffer is just an in-memory buffer, but it is still polite
to
// close early
IOUtils.closeStream(buf);
}
}
});
{code}
I know that the IOException is being caught because it's clear that
{{markClosed()}} is being called from the value of the {{closeException}}
variable. That variable is {{null}} for RPC requests that do not have problems,
but is a {{java.io.EOFException}} for the particular RPC call that this JIRA is
regarding.
So the problem is almost certainly somewhere in this code:
{code}
synchronized (ipcStreams.out) {
if (shouldCloseConnection.get()) {
return;
}
if (LOG.isDebugEnabled()) {
LOG.debug(getName() + " sending #" + call.id);
}
// RpcRequestHeader + RpcRequest
ipcStreams.sendRequest(buf.toByteArray());
ipcStreams.flush();
}
{code})
> ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when
> RPC privacy is enabled
> --
>
> Key: YARN-6013
> URL: https://issues.apache.org/jira/browse/YARN-6013
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, yarn
>Affects Versions: 2.8.0
>Reporter: Steven Rand
>Priority: Critical
> Attachments: yarn-rm-log.txt
>
>
> When privacy is enabled for RPC (hadoop.rpc.protection = privacy),
> {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always)
> fails with an EOFException. I've reproduced this with Spark 2.0.2 built
> against latest branch-2.8 and with a simple distcp job on latest branch-2.8.
> Steps to reproduce using distcp:
> 1. Set hadoop.rpc.protection equal to privacy
> 2. Write data to HDFS. I did this with Spark as follows:
> {code}
> sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k,
> org.apache.commons.lang.RandomStringUtils.random(1024,
> "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData")
> {code}
> 3. Attempt to distcp that data to another location in HDFS. For example:
> {code}
> hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData
> hdfs:///tmp/testDataCopy
> {code}
> I observed this error in the ApplicationMaster's syslog:
> {code}
> 2016-12-19 19:13:50,097 INFO [eventHandlingThread]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer
> setup for JobId: job_1482189777425_0004, File:
> hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist
> 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1482189777425_0004: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit= knownNMs=3
> 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator]
> org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking
> ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after
> sleeping for
[jira] [Commented] (YARN-5641) Localizer leaves behind tarballs after container is complete
[ https://issues.apache.org/jira/browse/YARN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840052#comment-15840052 ] Jason Lowe commented on YARN-5641: -- The branch-2 patch looks reasonable to me. Nit: the new isAlive method should be private. When fixing this, please post the patch according to the recommended [patch naming conventions|https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch] so we can reopen the JIRA and get the QA bot to run the standard tests on it. > Localizer leaves behind tarballs after container is complete > > > Key: YARN-5641 > URL: https://issues.apache.org/jira/browse/YARN-5641 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5641.001.patch, YARN-5641.002.patch, > YARN-5641.003.patch, YARN-5641.004.patch, YARN-5641.005.patch, > YARN-5641.006.patch, YARN-5641.007.patch, YARN-5641.008.patch, > YARN-5641.009.patch, YARN-5641.009.patch, YARN-5641.010-b2.patch, > YARN-5641.010.patch > > > The localizer sometimes fails to clean up extracted tarballs leaving large > footprints that persist on the nodes indefinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5703) ReservationAgents are not correctly configured
[ https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840169#comment-15840169 ] Manikandan R commented on YARN-5703: I am interested in working on this. Shall I take this forward? Thanks, Mani > ReservationAgents are not correctly configured > -- > > Key: YARN-5703 > URL: https://issues.apache.org/jira/browse/YARN-5703 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Po >Assignee: Sean Po > > In AbstractReservationSystem, the method that instantiates a ReservationAgent > does not properly initialize it with the appropriate configuration because it > expects the ReservationAgent to implement Configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6013) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[
https://issues.apache.org/jira/browse/YARN-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840119#comment-15840119
]
Steven Rand commented on YARN-6013:
---
// I deleted my above comment because it was inaccurate.
Looked into this more today with a debugger. I still haven't figured out quite
what's going on, but thought it might be useful to update this with some more
information.
One of the first four bytes of the {{token}} variable returned by
{{saslClient.unwrap}} \[1\] is consistently negative. Therefore
{{DataInputStream#readInt}} thinks that the stream has prematurely ended, and
throws an {{EOFException}}, when it's called by {{Client#readResponse}} \[2\].
\[1\]
https://github.com/apache/hadoop/blob/branch-2.8.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslRpcClient.java#L610
\[2\]
https://github.com/apache/hadoop/blob/branch-2.8.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1156
> ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when
> RPC privacy is enabled
> --
>
> Key: YARN-6013
> URL: https://issues.apache.org/jira/browse/YARN-6013
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, yarn
>Affects Versions: 2.8.0
>Reporter: Steven Rand
>Priority: Critical
> Attachments: yarn-rm-log.txt
>
>
> When privacy is enabled for RPC (hadoop.rpc.protection = privacy),
> {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always)
> fails with an EOFException. I've reproduced this with Spark 2.0.2 built
> against latest branch-2.8 and with a simple distcp job on latest branch-2.8.
> Steps to reproduce using distcp:
> 1. Set hadoop.rpc.protection equal to privacy
> 2. Write data to HDFS. I did this with Spark as follows:
> {code}
> sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k,
> org.apache.commons.lang.RandomStringUtils.random(1024,
> "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData")
> {code}
> 3. Attempt to distcp that data to another location in HDFS. For example:
> {code}
> hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData
> hdfs:///tmp/testDataCopy
> {code}
> I observed this error in the ApplicationMaster's syslog:
> {code}
> 2016-12-19 19:13:50,097 INFO [eventHandlingThread]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer
> setup for JobId: job_1482189777425_0004, File:
> hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist
> 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1482189777425_0004: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit= knownNMs=3
> 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator]
> org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking
> ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after
> sleeping for 3ms.
> java.io.EOFException: End of File Exception between local host is:
> "/"; destination host is: "":8030;
> : java.io.EOFException; For more details see:
> http://wiki.apache.org/hadoop/EOFException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486)
> at org.apache.hadoop.ipc.Client.call(Client.java:1428)
> at org.apache.hadoop.ipc.Client.call(Client.java:1338)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy80.allocate(Unknown Source)
> at
>
[jira] [Created] (YARN-6125) The application attempt's diagnostic message should have a maximum size
Daniel Templeton created YARN-6125: -- Summary: The application attempt's diagnostic message should have a maximum size Key: YARN-6125 URL: https://issues.apache.org/jira/browse/YARN-6125 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.7.0 Reporter: Daniel Templeton Assignee: Daniel Templeton Priority: Critical We've found through experience that the diagnostic message can grow unbounded. I've seen attempts that have diagnostic messages over 1MB. Since the message is stored in the state store, it's a bad idea to allow the message to grow unbounded. Instead, there should be a property that sets a maximum size on the message. I suspect that some of the ZK state store issues we've seen in the past were due to the size of the diagnostic messages and not to the size of the classpath, as is the current prevailing opinion. An open question is how best to prune the message once it grows too large. Should we # truncate the tail, # truncate the head, # truncate the middle, # add another property to make the behavior selectable, or # none of the above? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-3053) [Security] Review and implement authentication in ATS v.2
[ https://issues.apache.org/jira/browse/YARN-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840182#comment-15840182 ] Varun Saxena edited comment on YARN-3053 at 1/26/17 6:34 PM: - [~jianhe], first of all whatever I mentioned above is based on how I think off-app clients would work. We haven't yet finalized and brainstormed design of off-app clients because its not in immediate scope. For app collectors I agree with you that we do not need to store tokens in state store. By off-app clients I mean frameworks like Oozie or Hive publishing their entities i.e. some info before and after executing a workflow or a set of queries which does not necessarily fit within the scope of individiual AMs'. In other words, information which spans multiple YARN applications and hence cannot be published by individual AMs'. For such clients, collector address cannot be pushed like we do via RM-AM protocol for individual AMs'. So most probably clients will have to get this info from YARN (first when client comes up and later on connection loss if collector goes down). This query will most probably go to RM. And this communication needs to be kerberos authenticanted. We can probably open up a channel to pull a token as well from RM (after off-app collector reports it to RM via NM). But an easier way would be to get it from collector via an explicit get delegation token API. Because such clients would anyways need to do kerberos authentication. If we do so, we may need to store and recover tokens. Anyways off-app clients require a little more thought. I can think of ways of avoiding storing token even in this case but that may make other parts more complex. Let us discuss and handle use-cases of off-app clients once we start working on them. bq. I'm not sure the original collector design had accounted for unmanaged AM in general case Yes it hasn't been accounted for. We currently launch an app collector on a NM where AM container is launched. So we do not launch any app collector for unmanaged AM. Have raised YARN-6121 for that. was (Author: varun_saxena): [~jianhe], first of all whatever I mentioned above is based on how I think off-app clients would work. For app collectors I agree with you that we do not need to store tokens in state store. We haven't yet finalized and brainstormed design of off-app clients because its not in immediate scope. By off-app clients I mean frameworks like Oozie or Hive publishing their entities i.e. some info before and after executing a workflow or a set of queries which does not necessarily fit within the scope of individiual AMs'. In other words, information which spans multiple YARN applications and hence cannot be published by individual AMs'. For such clients, collector address cannot be pushed like we do via RM-AM protocol for individual AMs'. So most probably clients will have to get this info from YARN (first when client comes up and later on connection loss if collector goes down). This query will most probably go to RM. And this communication needs to be kerberos authenticanted. We can probably open up a channel to pull a token as well from RM (after off-app collector reports it to RM via NM). But an easier way would be to get it from collector via an explicit get delegation token API. Because such clients would anyways need to do kerberos authentication. If we do so, we may need to store and recover tokens. Anyways off-app clients require a little more thought. I can think of ways of avoiding storing token even in this case but that may make other parts more complex. Let us discuss and handle use-cases of off-app clients once we start working on them. bq. I'm not sure the original collector design had accounted for unmanaged AM in general case Yes it hasn't been accounted for. We currently launch an app collector on a NM where AM container is launched. So we do not launch any app collector for unmanaged AM. Have raised YARN-6121 for that. > [Security] Review and implement authentication in ATS v.2 > - > > Key: YARN-3053 > URL: https://issues.apache.org/jira/browse/YARN-3053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: ATSv2Authentication(draft).pdf > > > Per design in YARN-2928, we want to evaluate and review the system for > security, and ensure proper security in the system. > This includes proper authentication, token management, access control, and > any other relevant security aspects. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe,
[jira] [Commented] (YARN-3053) [Security] Review and implement authentication in ATS v.2
[ https://issues.apache.org/jira/browse/YARN-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840182#comment-15840182 ] Varun Saxena commented on YARN-3053: [~jianhe], first of all whatever I mentioned above is based on how I think off-app clients would work. For app collectors I agree with you that we do not need to store tokens in state store. We haven't yet finalized and brainstormed design of off-app clients because its not in immediate scope. By off-app clients I mean frameworks like Oozie or Hive publishing their entities i.e. some info before and after executing a workflow or a set of queries which does not necessarily fit within the scope of individiual AMs'. In other words, information which spans multiple YARN applications and hence cannot be published by individual AMs'. For such clients, collector address cannot be pushed like we do via RM-AM protocol for individual AMs'. So most probably clients will have to get this info from YARN (first when client comes up and later on connection loss if collector goes down). This query will most probably go to RM. And this communication needs to be kerberos authenticanted. We can probably open up a channel to pull a token as well from RM (after off-app collector reports it to RM via NM). But an easier way would be to get it from collector via an explicit get delegation token API. Because such clients would anyways need to do kerberos authentication. If we do so, we may need to store and recover tokens. Anyways off-app clients require a little more thought. I can think of ways of avoiding storing token even in this case but that may make other parts more complex. Let us discuss and handle use-cases of off-app clients once we start working on them. bq. I'm not sure the original collector design had accounted for unmanaged AM in general case Yes it hasn't been accounted for. We currently launch an app collector on a NM where AM container is launched. So we do not launch any app collector for unmanaged AM. Have raised YARN-6121 for that. > [Security] Review and implement authentication in ATS v.2 > - > > Key: YARN-3053 > URL: https://issues.apache.org/jira/browse/YARN-3053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: ATSv2Authentication(draft).pdf > > > Per design in YARN-2928, we want to evaluate and review the system for > security, and ensure proper security in the system. > This includes proper authentication, token management, access control, and > any other relevant security aspects. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6013) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[
https://issues.apache.org/jira/browse/YARN-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840286#comment-15840286
]
Steven Rand commented on YARN-6013:
---
Also the reason why other RPC calls are working is evidently that they're not
encrypted, which seems worrisome in its own right. For example, the code that
receives an RPC response from the NameNode for a lease renewal request goes
from {{Client$Connection$PingInputStream#read}} to {{SocketInputStream#read}}
and never makes its way into {{SaslRpcClient}} code.
> ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when
> RPC privacy is enabled
> --
>
> Key: YARN-6013
> URL: https://issues.apache.org/jira/browse/YARN-6013
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, yarn
>Affects Versions: 2.8.0
>Reporter: Steven Rand
>Priority: Critical
> Attachments: yarn-rm-log.txt
>
>
> When privacy is enabled for RPC (hadoop.rpc.protection = privacy),
> {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always)
> fails with an EOFException. I've reproduced this with Spark 2.0.2 built
> against latest branch-2.8 and with a simple distcp job on latest branch-2.8.
> Steps to reproduce using distcp:
> 1. Set hadoop.rpc.protection equal to privacy
> 2. Write data to HDFS. I did this with Spark as follows:
> {code}
> sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k,
> org.apache.commons.lang.RandomStringUtils.random(1024,
> "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData")
> {code}
> 3. Attempt to distcp that data to another location in HDFS. For example:
> {code}
> hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData
> hdfs:///tmp/testDataCopy
> {code}
> I observed this error in the ApplicationMaster's syslog:
> {code}
> 2016-12-19 19:13:50,097 INFO [eventHandlingThread]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer
> setup for JobId: job_1482189777425_0004, File:
> hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist
> 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1482189777425_0004: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit= knownNMs=3
> 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator]
> org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking
> ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after
> sleeping for 3ms.
> java.io.EOFException: End of File Exception between local host is:
> "/"; destination host is: "":8030;
> : java.io.EOFException; For more details see:
> http://wiki.apache.org/hadoop/EOFException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486)
> at org.apache.hadoop.ipc.Client.call(Client.java:1428)
> at org.apache.hadoop.ipc.Client.call(Client.java:1338)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy80.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
>
[jira] [Commented] (YARN-6123) [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues added or removed.
[
https://issues.apache.org/jira/browse/YARN-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840293#comment-15840293
]
Hadoop QA commented on YARN-6123:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 6 new + 123 unchanged - 0 fixed = 129 total (was 123)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 9s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6123 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12849549/YARN-6123.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b33511e8b566 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f85b74c |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14759/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14759/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14759/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14759/console
[jira] [Commented] (YARN-4975) Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues
[
https://issues.apache.org/jira/browse/YARN-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840246#comment-15840246
]
Hudson commented on YARN-4975:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11178 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11178/])
YARN-4975. Fair Scheduler: exception thrown when a parent queue marked
(templedf: rev f85b74ccf9f1c1c1444cc00750b03468cbf40fb9)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
> Fair Scheduler: exception thrown when a parent queue marked 'parent' has
> configured child queues
>
>
> Key: YARN-4975
> URL: https://issues.apache.org/jira/browse/YARN-4975
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.2
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-4975.001.patch, YARN-4975.002.patch
>
>
> We upgraded our clusters to 2.7.2 from 2.4.1 and saw the following exception
> in RM logs :
> {code}
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException:
> Both and type="parent" found for queue root.adhoc which is
> unsupported
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.loadQueue(AllocationFileLoaderService.java:519)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:352)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1440)
> {code}
> From the exception, it looks like we've configured 'reservation', but we've
> not. The issue is that AllocationFileLoaderService#loadQueue assumes that a
> parent queue marked as 'type=parent' cannot have configured child queues.
> That can be a problem in cases where we mark a queue as 'parent' which has no
> configured child queues to start with, but we can add child queues later on.
> Also the exception message is kind of misleading since we haven't configured
> 'reservation'.
> How to reproduce:
> Run fair scheduler with following queue config:
> {code}
>
> 10
> 300
>
> 3
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6013) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[
https://issues.apache.org/jira/browse/YARN-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840244#comment-15840244
]
Steven Rand commented on YARN-6013:
---
The problem is that {{Client$IpcStreams#readResponse}} is trying to get the
length of the input stream _after_ it's been unwrapped. Before being unwrapped,
the input stream does in fact contain the length of the RPC message in its
first four bytes, as we see in SaslRpcClient#readNextRpcPacket():
https://github.com/apache/hadoop/blob/branch-2.8.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslRpcClient.java#L589.
However, the stream that we pass to {{saslClient.unwrap}} can't contain those
four bytes. From the
[javadoc|https://docs.oracle.com/javase/8/docs/api/javax/security/sasl/SaslClient.html#unwrap-byte:A-int-int-]:
"incoming is the contents of the SASL buffer as defined in RFC without
the leading four octet field that represents the length." And indeed, the
length of the incoming RPC is 3586 in the case I'm looking at, but then
{{token}} is only 3570 bytes when we pass it to {{saslClient.unwrap}}, so the
first four bytes are definitely gone. (I'm not sure what the other 12 missing
bytes are though.)
Also, from what I can tell of the [SASL
code|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/com/sun/security/sasl/digest/DigestMD5Base.java],
the {{unwrap}} implementation won't put the 4-byte header back on.
So it doesn't make sense to call {{int length = in.readInt();}} on an unwrapped
stream, as the first four bytes of that do not contain its length. I can submit
a patch for this.
> ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when
> RPC privacy is enabled
> --
>
> Key: YARN-6013
> URL: https://issues.apache.org/jira/browse/YARN-6013
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, yarn
>Affects Versions: 2.8.0
>Reporter: Steven Rand
>Priority: Critical
> Attachments: yarn-rm-log.txt
>
>
> When privacy is enabled for RPC (hadoop.rpc.protection = privacy),
> {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always)
> fails with an EOFException. I've reproduced this with Spark 2.0.2 built
> against latest branch-2.8 and with a simple distcp job on latest branch-2.8.
> Steps to reproduce using distcp:
> 1. Set hadoop.rpc.protection equal to privacy
> 2. Write data to HDFS. I did this with Spark as follows:
> {code}
> sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k,
> org.apache.commons.lang.RandomStringUtils.random(1024,
> "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData")
> {code}
> 3. Attempt to distcp that data to another location in HDFS. For example:
> {code}
> hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData
> hdfs:///tmp/testDataCopy
> {code}
> I observed this error in the ApplicationMaster's syslog:
> {code}
> 2016-12-19 19:13:50,097 INFO [eventHandlingThread]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer
> setup for JobId: job_1482189777425_0004, File:
> hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist
> 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1482189777425_0004: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit= knownNMs=3
> 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator]
> org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking
> ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after
> sleeping for 3ms.
> java.io.EOFException: End of File Exception between local host is:
> "/"; destination host is: "":8030;
> : java.io.EOFException; For more details see:
> http://wiki.apache.org/hadoop/EOFException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
[jira] [Updated] (YARN-6123) [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues added or removed.
[ https://issues.apache.org/jira/browse/YARN-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6123: - Attachment: YARN-6123.002.patch > [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated > when childQueues added or removed. > --- > > Key: YARN-6123 > URL: https://issues.apache.org/jira/browse/YARN-6123 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6123.001.patch, YARN-6123.002.patch > > > YARN-5864 added queue ordering policy to ParentQueue, we need to make sure > queues of QueueOrderingPolicy will be updated when any changes made for child > queues. > We need to add a test to make sure it works. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-3053) [Security] Review and implement authentication in ATS v.2
[ https://issues.apache.org/jira/browse/YARN-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840182#comment-15840182 ] Varun Saxena edited comment on YARN-3053 at 1/26/17 6:38 PM: - [~jianhe], first of all whatever I mentioned above is based on how I think off-app clients would work. We haven't yet finalized and brainstormed design of off-app clients because its not in immediate scope. For app collectors I agree with you that we do not need to store tokens in state store. By off-app clients I mean frameworks like Oozie or Hive publishing their entities i.e. some info before and after executing a workflow or a set of queries which does not necessarily fit within the scope of individiual AMs'. In other words, information which spans multiple YARN applications and hence cannot be published by individual AMs'. For such clients, collector address cannot be pushed like we do via RM-AM protocol for individual AMs'. So most probably clients will have to get this info from YARN (first when client comes up and later on connection loss if collector goes down). This query will most probably go to RM. And this communication needs to be kerberos authenticanted. We can probably open up a channel to pull a token as well from RM (after off-app collector reports it to RM via NM). But an easier way would be to get it from collector via an explicit get delegation token API. Because such clients would anyways need to do kerberos authentication. If we do so, we may need to store and recover tokens. Anyways off-app clients require a little more thought. I can think of ways of avoiding storing token even in this case but that may make other parts more complex. Let us discuss and handle use-cases of off-app clients once we start working on them. Security design for off app clients would depend largely on how we decide on implementing them. bq. I'm not sure the original collector design had accounted for unmanaged AM in general case Yes it hasn't been accounted for. We currently launch an app collector on a NM where AM container is launched. So we do not launch any app collector for unmanaged AM. Have raised YARN-6121 for that. was (Author: varun_saxena): [~jianhe], first of all whatever I mentioned above is based on how I think off-app clients would work. We haven't yet finalized and brainstormed design of off-app clients because its not in immediate scope. For app collectors I agree with you that we do not need to store tokens in state store. By off-app clients I mean frameworks like Oozie or Hive publishing their entities i.e. some info before and after executing a workflow or a set of queries which does not necessarily fit within the scope of individiual AMs'. In other words, information which spans multiple YARN applications and hence cannot be published by individual AMs'. For such clients, collector address cannot be pushed like we do via RM-AM protocol for individual AMs'. So most probably clients will have to get this info from YARN (first when client comes up and later on connection loss if collector goes down). This query will most probably go to RM. And this communication needs to be kerberos authenticanted. We can probably open up a channel to pull a token as well from RM (after off-app collector reports it to RM via NM). But an easier way would be to get it from collector via an explicit get delegation token API. Because such clients would anyways need to do kerberos authentication. If we do so, we may need to store and recover tokens. Anyways off-app clients require a little more thought. I can think of ways of avoiding storing token even in this case but that may make other parts more complex. Let us discuss and handle use-cases of off-app clients once we start working on them. bq. I'm not sure the original collector design had accounted for unmanaged AM in general case Yes it hasn't been accounted for. We currently launch an app collector on a NM where AM container is launched. So we do not launch any app collector for unmanaged AM. Have raised YARN-6121 for that. > [Security] Review and implement authentication in ATS v.2 > - > > Key: YARN-3053 > URL: https://issues.apache.org/jira/browse/YARN-3053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: ATSv2Authentication(draft).pdf > > > Per design in YARN-2928, we want to evaluate and review the system for > security, and ensure proper security in the system. > This includes proper authentication, token management, access control, and > any other relevant security aspects. -- This message was sent by Atlassian JIRA
[jira] [Comment Edited] (YARN-5534) Allow whitelisted volume mounts
[
https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15835291#comment-15835291
]
Daniel Templeton edited comment on YARN-5534 at 1/26/17 6:56 PM:
-
Thanks for updating the patch, [~luhuichun]. The new patch matches more with
what I had in mind. There are still a couple of issues:
* In {{YarnConfiguration}}, you should add a Javadoc comment for the new
property
* In {{DLCR.isArbitraryMount()}}, instead of the _for_ loop, you should use a
_foreach_.
* {{DSCR.isArbitraryMount()}} always returns {{false}}. {code}File child =
new File(mount);
for (int i = 0; i < whiteList.length; i++){
File parent = new File(mount);
if (isSubDirectory(parent, child)){
return false;
}
}{code} It's always true that {{parent.equals(child)}}, so
{{isSubdirectory()}} will always return true.
* You should parse the white list property once and store it instead of doing
it every time the method is called
* Instead of using {{String.split()}}, you could use {{Pattern.split()}} to
allow for whitespace, making it a little more user friendly
* Look at
http://stackoverflow.com/questions/26530445/compare-directories-to-check-if-one-is-sub-directory-of-another
for ideas of how to do the subdirectory check more efficiently. I like the
{{Set}} idea.
* In {{DLCR.isSubdirectory()}}, the naming around _child_, _parent_, and
_parentFile_ is pretty confusing.
* In {{DLCR.isSubdirectory()}}, printing the stack trace is a bad idea. Do
something more useful. At a minimum, log the exception instead of printing the
stack trace to stderr.
* You should have a space before the curly brace throughout, e.g. {code} if
(parent.equals(parentFile)){ {code}
was (Author: templedf):
Thanks for updating the patch, [~luhuichun]. The new patch matches more with
what I had in mind. There are still a couple of issues:
* In {{YarnConfiguration}}, you should add a Javadoc comment for the new
property
* In {{DLCR.isArbitraryMount()}}, instead of the _for_ loop, you should use a
_foreach_.
* {{DSCR.isArbitraryMount()}} always returns {{false}}. {code}File child =
new File(mount);
for (int i = 0; i < whiteList.length; i++){
File parent = new File(mount);
if (isSubDirectory(parent, child)){
return false;
}
}{code} It's always true that {{parent.equals(child)}}, so
{{isSubdirectory()}} will always return true.
* You should parse the white list property once and store it instead of doing
it every time the method is called
* Instead of using {{String.split()}}, you could use {{Pattern.split()}} to
allow for whitespace, making it a little more user friendly
* Look at
http://stackoverflow.com/questions/26530445/compare-directories-to-check-if-one-is-sub-directory-of-another
for ideas of how to do the subdirectory check more efficiently. I like the
{{Set}} idea.
* In {{DLCR.isSubdirectory()}}, the naming around _child_, _parent_, and
_parentFile_ is pretty confusing.
* In {{DLCR.isSubdirectory()}}, printing the stack trace in a bad idea. Do
something more useful. At a minimum, log the exception instead of printing it
in stderr.
* You should have a space before the curly brace throughout, e.g. {code} if
(parent.equals(parentFile)){ {code}
> Allow whitelisted volume mounts
>
>
> Key: YARN-5534
> URL: https://issues.apache.org/jira/browse/YARN-5534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: luhuichun
>Assignee: luhuichun
> Attachments: YARN-5534.001.patch, YARN-5534.002.patch
>
>
> Introduction
> Mounting files or directories from the host is one way of passing
> configuration and other information into a docker container.
> We could allow the user to set a list of mounts in the environment of
> ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2).
> These would be mounted read-only to the specified target locations. This has
> been resolved in YARN-4595
> 2.Problem Definition
> Bug mounting arbitrary volumes into a Docker container can be a security risk.
> 3.Possible solutions
> one approach to provide safe mounts is to allow the cluster administrator to
> configure a set of parent directories as white list mounting directories.
> Add a property named yarn.nodemanager.volume-mounts.white-list, when
> container executor do mount checking, only the allowed directories or
> sub-directories can be mounted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6121) Launch app collectors for unmanaged AMs'
[ https://issues.apache.org/jira/browse/YARN-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6121: --- Description: Currently an app collector is launched whenever an AM container is launched on a NM. This means for an unmanaged AM, app collector is never launched. > Launch app collectors for unmanaged AMs' > > > Key: YARN-6121 > URL: https://issues.apache.org/jira/browse/YARN-6121 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Varun Saxena > > Currently an app collector is launched whenever an AM container is launched > on a NM. This means for an unmanaged AM, app collector is never launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5641) Localizer leaves behind tarballs after container is complete
[ https://issues.apache.org/jira/browse/YARN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-5641: -- Attachment: YARN-5641-branch-2.011.patch Changed isAlive() method to private > Localizer leaves behind tarballs after container is complete > > > Key: YARN-5641 > URL: https://issues.apache.org/jira/browse/YARN-5641 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5641.001.patch, YARN-5641.002.patch, > YARN-5641.003.patch, YARN-5641.004.patch, YARN-5641.005.patch, > YARN-5641.006.patch, YARN-5641.007.patch, YARN-5641.008.patch, > YARN-5641.009.patch, YARN-5641.009.patch, YARN-5641.010-b2.patch, > YARN-5641.010.patch, YARN-5641-branch-2.011.patch > > > The localizer sometimes fails to clean up extracted tarballs leaving large > footprints that persist on the nodes indefinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6125) The application attempt's diagnostic message should have a maximum size
[ https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840265#comment-15840265 ] Jason Lowe commented on YARN-6125: -- For the huge examples that have been encountered so far, what would have worked best for them? Are they simply a gigantic stacktrace, an accumulation of independent diagnostic messages, or potentially recurring, redundant messages for the same error? I normally would tend to lean towards preserving the tail end of the message with the assumption that the most recent error would be logged there, but of course there could be cascading errors and the beginning would be better. That's why I'm hoping the real-world examples help shape the direction here. I'd rather not add yet another config that either nobody sets or knows how to set correctly. If we do add a config then the next question is whether that config should be app-specific (e.g.: app framework A's diagnostic approach works best with preserving the end, but preserving the beginning is better for B, etc.). > The application attempt's diagnostic message should have a maximum size > --- > > Key: YARN-6125 > URL: https://issues.apache.org/jira/browse/YARN-6125 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > > We've found through experience that the diagnostic message can grow > unbounded. I've seen attempts that have diagnostic messages over 1MB. Since > the message is stored in the state store, it's a bad idea to allow the > message to grow unbounded. Instead, there should be a property that sets a > maximum size on the message. > I suspect that some of the ZK state store issues we've seen in the past were > due to the size of the diagnostic messages and not to the size of the > classpath, as is the current prevailing opinion. > An open question is how best to prune the message once it grows too large. > Should we > # truncate the tail, > # truncate the head, > # truncate the middle, > # add another property to make the behavior selectable, or > # none of the above? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6125) The application attempt's diagnostic message should have a maximum size
[ https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6125: --- Assignee: Andras Piros (was: Daniel Templeton) > The application attempt's diagnostic message should have a maximum size > --- > > Key: YARN-6125 > URL: https://issues.apache.org/jira/browse/YARN-6125 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Daniel Templeton >Assignee: Andras Piros >Priority: Critical > > We've found through experience that the diagnostic message can grow > unbounded. I've seen attempts that have diagnostic messages over 1MB. Since > the message is stored in the state store, it's a bad idea to allow the > message to grow unbounded. Instead, there should be a property that sets a > maximum size on the message. > I suspect that some of the ZK state store issues we've seen in the past were > due to the size of the diagnostic messages and not to the size of the > classpath, as is the current prevailing opinion. > An open question is how best to prune the message once it grows too large. > Should we > # truncate the tail, > # truncate the head, > # truncate the middle, > # add another property to make the behavior selectable, or > # none of the above? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5703) ReservationAgents are not correctly configured
[ https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-5703: Assignee: Manikandan R (was: Sean Po) > ReservationAgents are not correctly configured > -- > > Key: YARN-5703 > URL: https://issues.apache.org/jira/browse/YARN-5703 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Po >Assignee: Manikandan R > > In AbstractReservationSystem, the method that instantiates a ReservationAgent > does not properly initialize it with the appropriate configuration because it > expects the ReservationAgent to implement Configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5703) ReservationAgents are not correctly configured
[ https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842308#comment-15842308 ] Naganarasimha G R commented on YARN-5703: - Added [~maniraj...@gmail.com] to the contirbutor's list and assigned the jira to him ! Hope to see some Patch if you have approach in your mind ! > ReservationAgents are not correctly configured > -- > > Key: YARN-5703 > URL: https://issues.apache.org/jira/browse/YARN-5703 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Po >Assignee: Manikandan R > > In AbstractReservationSystem, the method that instantiates a ReservationAgent > does not properly initialize it with the appropriate configuration because it > expects the ReservationAgent to implement Configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5966) AMRMClient changes to support ExecutionType update
[
https://issues.apache.org/jira/browse/YARN-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839455#comment-15839455
]
Hadoop QA commented on YARN-5966:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 0s{color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 5 new + 35 unchanged -
0 fixed = 40 total (was 35) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 10 new + 149 unchanged - 4 fixed = 159 total (was 153) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
26s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 41m
54s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
41s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5966 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12849469/YARN-5966.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 27bacf48e840 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 425a7e5 |
| Default Java |
[jira] [Commented] (YARN-6123) [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues added or removed.
[ https://issues.apache.org/jira/browse/YARN-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840183#comment-15840183 ] Wangda Tan commented on YARN-6123: -- Thanks [~sunilg] for reviewing it, For 1. I think it is safe, since now assignmentIterator is called under CS's readlock, and reinitialize queue needs CS's writeLock. For 2. Addressed. > [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated > when childQueues added or removed. > --- > > Key: YARN-6123 > URL: https://issues.apache.org/jira/browse/YARN-6123 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6123.001.patch, YARN-6123.002.patch > > > YARN-5864 added queue ordering policy to ParentQueue, we need to make sure > queues of QueueOrderingPolicy will be updated when any changes made for child > queues. > We need to add a test to make sure it works. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6123) [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues added or removed.
[ https://issues.apache.org/jira/browse/YARN-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840218#comment-15840218 ] Sunil G commented on YARN-6123: --- Latest patch looks good to me. Pending jenkins. > [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated > when childQueues added or removed. > --- > > Key: YARN-6123 > URL: https://issues.apache.org/jira/browse/YARN-6123 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6123.001.patch, YARN-6123.002.patch > > > YARN-5864 added queue ordering policy to ParentQueue, we need to make sure > queues of QueueOrderingPolicy will be updated when any changes made for child > queues. > We need to add a test to make sure it works. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-5641) Localizer leaves behind tarballs after container is complete
[ https://issues.apache.org/jira/browse/YARN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reopened YARN-5641: -- Reopening to run Jenkins on the branch-2 patch. > Localizer leaves behind tarballs after container is complete > > > Key: YARN-5641 > URL: https://issues.apache.org/jira/browse/YARN-5641 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5641.001.patch, YARN-5641.002.patch, > YARN-5641.003.patch, YARN-5641.004.patch, YARN-5641.005.patch, > YARN-5641.006.patch, YARN-5641.007.patch, YARN-5641.008.patch, > YARN-5641.009.patch, YARN-5641.009.patch, YARN-5641.010-b2.patch, > YARN-5641.010.patch, YARN-5641-branch-2.011.patch > > > The localizer sometimes fails to clean up extracted tarballs leaving large > footprints that persist on the nodes indefinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5966) AMRMClient changes to support ExecutionType update
[
https://issues.apache.org/jira/browse/YARN-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840438#comment-15840438
]
Subru Krishnan commented on YARN-5966:
--
Thanks [~asuresh] for addressing all my feedback. +1 on the latest patch
pending Yetus javac/checkstyle warnings fix.
> AMRMClient changes to support ExecutionType update
> --
>
> Key: YARN-5966
> URL: https://issues.apache.org/jira/browse/YARN-5966
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5966.001.patch, YARN-5966.002.patch,
> YARN-5966.003.patch, YARN-5966.004.patch, YARN-5966.005.patch,
> YARN-5966.006.patch, YARN-5966.wip.001.patch
>
>
> {{AMRMClient}} changes to support change of container ExecutionType
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5734) OrgQueue for easy CapacityScheduler queue configuration management
[
https://issues.apache.org/jira/browse/YARN-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840432#comment-15840432
]
Wangda Tan commented on YARN-5734:
--
Hi [~jhung],
bq. With this in mind do you still think AdminService is the right place to put
the change configuration functionality?
I would still prefer to use AdminService, we can add different logic to check
ACLs inside AdminService. It is still better than adding them to
ClientRMService.
bq. If we make MutableConfigurationManager part of CS only, the
ClientRMService/AdminService still needs to access it somehow.
I think we can make AdminService to call CS directly (like adding a method to
CS like {{updateCSConfig}}), and inside CS we will check and reject the
request. Changing the global provide-class looks more risks to me, since all
YARN components are depended upon that. It's better to limit logics inside CS.
> OrgQueue for easy CapacityScheduler queue configuration management
> --
>
> Key: YARN-5734
> URL: https://issues.apache.org/jira/browse/YARN-5734
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Min Shen
>Assignee: Min Shen
> Attachments: OrgQueue_API-Based_Config_Management_v1.pdf,
> OrgQueueAPI-BasedSchedulerConfigurationManagement_v2.pdf,
> OrgQueue_Design_v0.pdf, YARN-5734-YARN-5734.001.patch
>
>
> The current xml based configuration mechanism in CapacityScheduler makes it
> very inconvenient to apply any changes to the queue configurations. We saw 2
> main drawbacks in the file based configuration mechanism:
> # This makes it very inconvenient to automate queue configuration updates.
> For example, in our cluster setup, we leverage the queue mapping feature from
> YARN-2411 to route users to their dedicated organization queues. It could be
> extremely cumbersome to keep updating the config file to manage the very
> dynamic mapping between users to organizations.
> # Even a user has the admin permission on one specific queue, that user is
> unable to make any queue configuration changes to resize the subqueues,
> changing queue ACLs, or creating new queues. All these operations need to be
> performed in a centralized manner by the cluster administrators.
> With these current limitations, we realized the need of a more flexible
> configuration mechanism that allows queue configurations to be stored and
> managed more dynamically. We developed the feature internally at LinkedIn
> which introduces the concept of MutableConfigurationProvider. What it
> essentially does is to provide a set of configuration mutation APIs that
> allows queue configurations to be updated externally with a set of REST APIs.
> When performing the queue configuration changes, the queue ACLs will be
> honored, which means only queue administrators can make configuration changes
> to a given queue. MutableConfigurationProvider is implemented as a pluggable
> interface, and we have one implementation of this interface which is based on
> Derby embedded database.
> This feature has been deployed at LinkedIn's Hadoop cluster for a year now,
> and have gone through several iterations of gathering feedbacks from users
> and improving accordingly. With this feature, cluster administrators are able
> to automate lots of thequeue configuration management tasks, such as setting
> the queue capacities to adjust cluster resources between queues based on
> established resource consumption patterns, or managing updating the user to
> queue mappings. We have attached our design documentation with this ticket
> and would like to receive feedbacks from the community regarding how to best
> integrate it with the latest version of YARN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3053) [Security] Review and implement authentication in ATS v.2
[ https://issues.apache.org/jira/browse/YARN-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840442#comment-15840442 ] Jian He commented on YARN-3053: --- Yeah, makes sense to me. Let's move the discussion to YARN-6121 for off-apps. I think we have general consensus here for managed AMs. Would you like to update the design doc and may be open sub-jiras and start the development ? > [Security] Review and implement authentication in ATS v.2 > - > > Key: YARN-3053 > URL: https://issues.apache.org/jira/browse/YARN-3053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: ATSv2Authentication(draft).pdf > > > Per design in YARN-2928, we want to evaluate and review the system for > security, and ensure proper security in the system. > This includes proper authentication, token management, access control, and > any other relevant security aspects. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6125) The application attempt's diagnostic message should have a maximum size
[ https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840473#comment-15840473 ] Jason Lowe commented on YARN-6125: -- So I assume the tail end of the diagnostics would have been just as good as the first? If so, my vote is to roll this out initially with preserving the tail end. We'll have to come up with a reasonable default limit so most stacktraces/errors fit to avoid making things hard to debug by default. > The application attempt's diagnostic message should have a maximum size > --- > > Key: YARN-6125 > URL: https://issues.apache.org/jira/browse/YARN-6125 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Daniel Templeton >Assignee: Andras Piros >Priority: Critical > > We've found through experience that the diagnostic message can grow > unbounded. I've seen attempts that have diagnostic messages over 1MB. Since > the message is stored in the state store, it's a bad idea to allow the > message to grow unbounded. Instead, there should be a property that sets a > maximum size on the message. > I suspect that some of the ZK state store issues we've seen in the past were > due to the size of the diagnostic messages and not to the size of the > classpath, as is the current prevailing opinion. > An open question is how best to prune the message once it grows too large. > Should we > # truncate the tail, > # truncate the head, > # truncate the middle, > # add another property to make the behavior selectable, or > # none of the above? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6125) The application attempt's diagnostic message should have a maximum size
[ https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840490#comment-15840490 ] Daniel Templeton commented on YARN-6125: Agreed. I don't think we need to be too draconian on the limit. 64k seems like a reasonable limit. It's big enough that nothing reasonable should overrun it, but a hell of a lot better than 1MB. > The application attempt's diagnostic message should have a maximum size > --- > > Key: YARN-6125 > URL: https://issues.apache.org/jira/browse/YARN-6125 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Daniel Templeton >Assignee: Andras Piros >Priority: Critical > > We've found through experience that the diagnostic message can grow > unbounded. I've seen attempts that have diagnostic messages over 1MB. Since > the message is stored in the state store, it's a bad idea to allow the > message to grow unbounded. Instead, there should be a property that sets a > maximum size on the message. > I suspect that some of the ZK state store issues we've seen in the past were > due to the size of the diagnostic messages and not to the size of the > classpath, as is the current prevailing opinion. > An open question is how best to prune the message once it grows too large. > Should we > # truncate the tail, > # truncate the head, > # truncate the middle, > # add another property to make the behavior selectable, or > # none of the above? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5641) Localizer leaves behind tarballs after container is complete
[
https://issues.apache.org/jira/browse/YARN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840519#comment-15840519
]
Hadoop QA commented on YARN-5641:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
13s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
20s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
12s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 12s{color}
| {color:red} root-jdk1.8.0_121 with JDK v1.8.0_121 generated 1 new + 879
unchanged - 1 fixed = 880 total (was 880) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
57s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 57s{color}
| {color:red} root-jdk1.7.0_121 with JDK v1.7.0_121 generated 1 new + 974
unchanged - 1 fixed = 975 total (was 975) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 36s{color} | {color:orange} root: The patch generated 6 new + 101 unchanged
- 0 fixed = 107 total (was 101) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
5s{color} | {color:green} hadoop-common in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
18s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed
with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_121 Failed junit tests | hadoop.ha.TestZKFailoverController |
| |
[jira] [Commented] (YARN-6059) Update paused container state in the state store
[
https://issues.apache.org/jira/browse/YARN-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840805#comment-15840805
]
Hadoop QA commented on YARN-6059:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-6059 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6059 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12849619/YARN-5216-YARN-6059.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14764/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Update paused container state in the state store
>
>
> Key: YARN-6059
> URL: https://issues.apache.org/jira/browse/YARN-6059
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Hitesh Sharma
>Assignee: Hitesh Sharma
> Attachments: YARN-5216-YARN-6059.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[
https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840808#comment-15840808
]
Hadoop QA commented on YARN-5258:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12849596/YARN-5258.005.patch |
| Optional Tests | asflicense mvnsite xml |
| uname | Linux b481443268ea 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7bc333a |
| modules | C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14763/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Document Use of Docker with LinuxContainerExecutor
> --
>
> Key: YARN-5258
> URL: https://issues.apache.org/jira/browse/YARN-5258
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: documentation
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Labels: oct16-easy
> Attachments: YARN-5258.001.patch, YARN-5258.002.patch,
> YARN-5258.003.patch, YARN-5258.004.patch, YARN-5258.005.patch
>
>
> There aren't currently any docs that explain how to configure Docker and all
> of its various options aside from reading all of the JIRAs. We need to
> document the configuration, use, and troubleshooting, along with helpful
> examples.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6061) Add a customized uncaughtexceptionhandler for critical threads in RM
[ https://issues.apache.org/jira/browse/YARN-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6061: --- Summary: Add a customized uncaughtexceptionhandler for critical threads in RM (was: Add a customized uncaughtexceptionhandler for critical threads) > Add a customized uncaughtexceptionhandler for critical threads in RM > > > Key: YARN-6061 > URL: https://issues.apache.org/jira/browse/YARN-6061 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6061.001.patch, YARN-6061.002.patch > > > There are several threads in fair scheduler. The thread will quit when there > is a runtime exception inside it. We should bring down the RM when that > happens. Otherwise, there may be some weird behavior in RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6126) Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:"
[
https://issues.apache.org/jira/browse/YARN-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840842#comment-15840842
]
Hadoop QA commented on YARN-6126:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
21s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6126 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12849610/YARN-6126.trunk.1.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d772eb8553ee 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7bc333a |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14765/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14765/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Obtaining app logs for Running application fails with "Unable to parse json
> from webservice. Error:"
>
>
> Key: YARN-6126
> URL: https://issues.apache.org/jira/browse/YARN-6126
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Sumana Sathish
>Assignee: Xuan Gong
> Attachments:
[jira] [Commented] (YARN-6125) The application attempt's diagnostic message should have a maximum size
[ https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840451#comment-15840451 ] Daniel Templeton commented on YARN-6125: In the case that I've gotten my hands on the message contents, it was a repeated stack trace from Spark, one for each of many, many retries. > The application attempt's diagnostic message should have a maximum size > --- > > Key: YARN-6125 > URL: https://issues.apache.org/jira/browse/YARN-6125 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Daniel Templeton >Assignee: Andras Piros >Priority: Critical > > We've found through experience that the diagnostic message can grow > unbounded. I've seen attempts that have diagnostic messages over 1MB. Since > the message is stored in the state store, it's a bad idea to allow the > message to grow unbounded. Instead, there should be a property that sets a > maximum size on the message. > I suspect that some of the ZK state store issues we've seen in the past were > due to the size of the diagnostic messages and not to the size of the > classpath, as is the current prevailing opinion. > An open question is how best to prune the message once it grows too large. > Should we > # truncate the tail, > # truncate the head, > # truncate the middle, > # add another property to make the behavior selectable, or > # none of the above? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5703) ReservationAgents are not correctly configured
[ https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840458#comment-15840458 ] Sean Po commented on YARN-5703: --- Hi [~maniraj...@gmail.com], thanks for your interest in this. Please feel free to work on this JIRA. > ReservationAgents are not correctly configured > -- > > Key: YARN-5703 > URL: https://issues.apache.org/jira/browse/YARN-5703 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Po >Assignee: Sean Po > > In AbstractReservationSystem, the method that instantiates a ReservationAgent > does not properly initialize it with the appropriate configuration because it > expects the ReservationAgent to implement Configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6059) Update paused container state in the state store
[ https://issues.apache.org/jira/browse/YARN-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Sharma updated YARN-6059: Attachment: YARN-5216-YARN-6059.001.patch > Update paused container state in the state store > > > Key: YARN-6059 > URL: https://issues.apache.org/jira/browse/YARN-6059 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Sharma >Assignee: Hitesh Sharma > Attachments: YARN-5216-YARN-6059.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6112) fsOpDurations.addUpdateCallDuration() should be independent to LOG level
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840753#comment-15840753
]
Hadoop QA commented on YARN-6112:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 46 unchanged - 1 fixed = 46 total (was 47) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m
14s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6112 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12849602/YARN-6112.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 166687af5eac 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 55c9f6d |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14762/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14762/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> fsOpDurations.addUpdateCallDuration() should be independent to LOG level
>
>
> Key: YARN-6112
> URL:
[jira] [Commented] (YARN-5641) Localizer leaves behind tarballs after container is complete
[ https://issues.apache.org/jira/browse/YARN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840535#comment-15840535 ] Eric Badger commented on YARN-5641: --- The test failures are unrelated. However, since I got rid of the waitFor() parameters (not supported in Java 7), there is an unused TimeUnit import. The warnings are from a piece of code that wasn't touched. > Localizer leaves behind tarballs after container is complete > > > Key: YARN-5641 > URL: https://issues.apache.org/jira/browse/YARN-5641 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5641.001.patch, YARN-5641.002.patch, > YARN-5641.003.patch, YARN-5641.004.patch, YARN-5641.005.patch, > YARN-5641.006.patch, YARN-5641.007.patch, YARN-5641.008.patch, > YARN-5641.009.patch, YARN-5641.009.patch, YARN-5641.010-b2.patch, > YARN-5641.010.patch, YARN-5641-branch-2.011.patch > > > The localizer sometimes fails to clean up extracted tarballs leaving large > footprints that persist on the nodes indefinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent
[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne reassigned YARN-5892:
Assignee: Eric Payne
> Capacity Scheduler: Support user-specific minimum user limit percent
>
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
>
>
> yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
> 25
>
>
>
> yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
> 75
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4997) Update fair scheduler to use pluggable auth provider
[ https://issues.apache.org/jira/browse/YARN-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4997: --- Fix Version/s: 2.9.0 > Update fair scheduler to use pluggable auth provider > > > Key: YARN-4997 > URL: https://issues.apache.org/jira/browse/YARN-4997 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Tao Jie > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-4997-001.patch, YARN-4997-002.patch, > YARN-4997-003.patch, YARN-4997-004.patch, YARN-4997-005.patch, > YARN-4997-006.patch, YARN-4997-007.patch, YARN-4997-008.patch, > YARN-4997-009.patch, YARN-4997-010.patch, YARN-4997-011.patch > > > Now that YARN-3100 has made the authorization pluggable, it should be > supported by the fair scheduler. YARN-3100 only updated the capacity > scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6112) fsOpDurations.addUpdateCallDuration() should be independent to LOG level
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840543#comment-15840543
]
Karthik Kambatla commented on YARN-6112:
Sorry for screwing it up. Absolutely, it is a mistake.
That said, maybe the code corresponding to measuring these durations should be
outside the writelock. Could you also update {{nodeUpdate}} similarly?
> fsOpDurations.addUpdateCallDuration() should be independent to LOG level
>
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6112.001.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6112) fsOpDurations.addUpdateCallDuration() should be independent to LOG level
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840585#comment-15840585
]
Yufei Gu commented on YARN-6112:
There is an another duration measurement outside of the function update()
(shows in following code), which did the measurement outside the writelock.
It's a bit surprise me, by looking at the original design in YARN-2352. There
are two because there is one more line after {{update()}}. We should remove one
of measurements since they are identical.
{code}
long start = getClock().getTime();
update();
long duration = getClock().getTime() - start;
fsOpDurations.addUpdateThreadRunDuration(duration);
{code}
> fsOpDurations.addUpdateCallDuration() should be independent to LOG level
>
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6112.001.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6112) fsOpDurations.addUpdateCallDuration() should be independent to LOG level
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840585#comment-15840585
]
Yufei Gu edited comment on YARN-6112 at 1/26/17 10:42 PM:
--
There is an another duration measurement outside of the function update()
(shows in following code), which did the measurement outside the writelock.
It's a bit surprise me, by looking at the original design in YARN-2352. There
are two because there is one more line after {{update()}}. We should remove one
of measurements since they are identical now. One of my concerns is that is it
OK to remove one of the metrics(updateThreadRun, updateCall)?
{code}
long start = getClock().getTime();
update();
long duration = getClock().getTime() - start;
fsOpDurations.addUpdateThreadRunDuration(duration);
{code}
was (Author: yufeigu):
There is an another duration measurement outside of the function update()
(shows in following code), which did the measurement outside the writelock.
It's a bit surprise me, by looking at the original design in YARN-2352. There
are two because there is one more line after {{update()}}. We should remove one
of measurements since they are identical.
{code}
long start = getClock().getTime();
update();
long duration = getClock().getTime() - start;
fsOpDurations.addUpdateThreadRunDuration(duration);
{code}
> fsOpDurations.addUpdateCallDuration() should be independent to LOG level
>
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6112.001.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6112) fsOpDurations.addUpdateCallDuration() should be independent to LOG level
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6112:
---
Attachment: YARN-6112.002.patch
The patch 002 moves measurement outside the writelock.
> fsOpDurations.addUpdateCallDuration() should be independent to LOG level
>
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6112.001.patch, YARN-6112.002.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6112) fsOpDurations.addUpdateCallDuration() should be independent to LOG level
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6112:
---
Attachment: (was: YARN-6112.003.patch)
> fsOpDurations.addUpdateCallDuration() should be independent to LOG level
>
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6112.001.patch, YARN-6112.002.patch,
> YARN-6112.003.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6112) fsOpDurations.addUpdateCallDuration() should be independent to LOG level
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6112:
---
Attachment: YARN-6112.003.patch
> fsOpDurations.addUpdateCallDuration() should be independent to LOG level
>
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6112.001.patch, YARN-6112.002.patch,
> YARN-6112.003.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6112) fsOpDurations.addUpdateCallDuration() should be independent to LOG level
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840655#comment-15840655
]
Yufei Gu commented on YARN-6112:
Patch 003:
Remove the updateCall metrics and related measurement in function {{update}}
since it is identical to updateThreadRun right now. We'll keep it in branch-2
as [~kasha] suggested.
> fsOpDurations.addUpdateCallDuration() should be independent to LOG level
>
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6112.001.patch, YARN-6112.002.patch,
> YARN-6112.003.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6126) Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:"
Xuan Gong created YARN-6126: --- Summary: Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:" Key: YARN-6126 URL: https://issues.apache.org/jira/browse/YARN-6126 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6126) Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:"
[ https://issues.apache.org/jira/browse/YARN-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-6126: Description: After YARN-6099, we can output aggregated log files info as well as local log files info if we enabled partial log aggregation. This breaks the original logic in logsCLI > Obtaining app logs for Running application fails with "Unable to parse json > from webservice. Error:" > > > Key: YARN-6126 > URL: https://issues.apache.org/jira/browse/YARN-6126 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > After YARN-6099, we can output aggregated log files info as well as local log > files info if we enabled partial log aggregation. This breaks the original > logic in logsCLI -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6126) Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:"
[
https://issues.apache.org/jira/browse/YARN-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840670#comment-15840670
]
Xuan Gong commented on YARN-6126:
-
It is because:
1) if partial log aggregation is enabled, the object ("containerLogsInfo") is
JSONArray type
2) if not, the object("containerLogsInfo") is JSONObject
We should handle both of them. Uploaded a patch to fix it.
It is hard to add a unit test, but do the manual test and verify it works fine.
> Obtaining app logs for Running application fails with "Unable to parse json
> from webservice. Error:"
>
>
> Key: YARN-6126
> URL: https://issues.apache.org/jira/browse/YARN-6126
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>
> After YARN-6099, we can output aggregated log files info as well as local log
> files info if we enabled partial log aggregation. This breaks the original
> logic in logsCLI
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6126) Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:"
[ https://issues.apache.org/jira/browse/YARN-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-6126: Attachment: YARN-6126.branch-2.1.patch > Obtaining app logs for Running application fails with "Unable to parse json > from webservice. Error:" > > > Key: YARN-6126 > URL: https://issues.apache.org/jira/browse/YARN-6126 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-6126.branch-2.1.patch > > > After YARN-6099, we can output aggregated log files info as well as local log > files info if we enabled partial log aggregation. This breaks the original > logic in logsCLI -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6126) Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:"
[ https://issues.apache.org/jira/browse/YARN-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-6126: Attachment: YARN-6126.trunk.1.patch > Obtaining app logs for Running application fails with "Unable to parse json > from webservice. Error:" > > > Key: YARN-6126 > URL: https://issues.apache.org/jira/browse/YARN-6126 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-6126.branch-2.1.patch, YARN-6126.trunk.1.patch > > > After YARN-6099, we can output aggregated log files info as well as local log > files info if we enabled partial log aggregation. This breaks the original > logic in logsCLI -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6118) Add javadoc for Resources.isNone
[ https://issues.apache.org/jira/browse/YARN-6118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840684#comment-15840684 ] Andres Perez commented on YARN-6118: I can't because I'm still not in the approved list for yarn > Add javadoc for Resources.isNone > > > Key: YARN-6118 > URL: https://issues.apache.org/jira/browse/YARN-6118 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Priority: Minor > Labels: newbie > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6126) Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:"
[ https://issues.apache.org/jira/browse/YARN-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-6126: Reporter: Sumana Sathish (was: Xuan Gong) > Obtaining app logs for Running application fails with "Unable to parse json > from webservice. Error:" > > > Key: YARN-6126 > URL: https://issues.apache.org/jira/browse/YARN-6126 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sumana Sathish >Assignee: Xuan Gong > Attachments: YARN-6126.branch-2.1.patch, YARN-6126.trunk.1.patch > > > After YARN-6099, we can output aggregated log files info as well as local log > files info if we enabled partial log aggregation. This breaks the original > logic in logsCLI -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4975) Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues
[
https://issues.apache.org/jira/browse/YARN-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840551#comment-15840551
]
Daniel Templeton commented on YARN-4975:
Nevermind. I pulled in YARN-4997 and YARN-6000, and now this patch applies to
branch-2 cleanly.
> Fair Scheduler: exception thrown when a parent queue marked 'parent' has
> configured child queues
>
>
> Key: YARN-4975
> URL: https://issues.apache.org/jira/browse/YARN-4975
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.2
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-4975.001.patch, YARN-4975.002.patch
>
>
> We upgraded our clusters to 2.7.2 from 2.4.1 and saw the following exception
> in RM logs :
> {code}
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException:
> Both and type="parent" found for queue root.adhoc which is
> unsupported
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.loadQueue(AllocationFileLoaderService.java:519)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:352)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1440)
> {code}
> From the exception, it looks like we've configured 'reservation', but we've
> not. The issue is that AllocationFileLoaderService#loadQueue assumes that a
> parent queue marked as 'type=parent' cannot have configured child queues.
> That can be a problem in cases where we mark a queue as 'parent' which has no
> configured child queues to start with, but we can add child queues later on.
> Also the exception message is kind of misleading since we haven't configured
> 'reservation'.
> How to reproduce:
> Run fair scheduler with following queue config:
> {code}
>
> 10
> 300
>
> 3
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6000) Make AllocationFileLoaderService.Listener public
[
https://issues.apache.org/jira/browse/YARN-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6000:
---
Fix Version/s: 2.9.0
> Make AllocationFileLoaderService.Listener public
>
>
> Key: YARN-6000
> URL: https://issues.apache.org/jira/browse/YARN-6000
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Tao Jie
>Assignee: Tao Jie
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-6000.001.patch
>
>
> We removed public modifier of {{AllocationFileLoaderService.Listener}} in
> YARN-4997 since it trigger a findbugs warning. However it breaks Hive code in
> {{FairSchedulerShim}}.
> {code}
> AllocationFileLoaderService allocsLoader = new AllocationFileLoaderService();
> allocsLoader.init(conf);
> allocsLoader.setReloadListener(new AllocationFileLoaderService.Listener()
> {
> @Override
> public void onReload(AllocationConfiguration allocs) {
> allocConf.set(allocs);
> }
> });
> try {
> allocsLoader.reloadAllocations();
> } catch (Exception ex) {
> throw new IOException("Failed to load queue allocations", ex);
> }
> if (allocConf.get() == null) {
> allocConf.set(new AllocationConfiguration(conf));
> }
> QueuePlacementPolicy queuePolicy = allocConf.get().getPlacementPolicy();
> if (queuePolicy != null) {
> requestedQueue = queuePolicy.assignAppToQueue(requestedQueue, userName);
> {code}
> As a result we should set the modifier back to public.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4975) Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues
[
https://issues.apache.org/jira/browse/YARN-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-4975:
---
Fix Version/s: 2.9.0
> Fair Scheduler: exception thrown when a parent queue marked 'parent' has
> configured child queues
>
>
> Key: YARN-4975
> URL: https://issues.apache.org/jira/browse/YARN-4975
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.2
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-4975.001.patch, YARN-4975.002.patch
>
>
> We upgraded our clusters to 2.7.2 from 2.4.1 and saw the following exception
> in RM logs :
> {code}
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException:
> Both and type="parent" found for queue root.adhoc which is
> unsupported
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.loadQueue(AllocationFileLoaderService.java:519)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:352)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1440)
> {code}
> From the exception, it looks like we've configured 'reservation', but we've
> not. The issue is that AllocationFileLoaderService#loadQueue assumes that a
> parent queue marked as 'type=parent' cannot have configured child queues.
> That can be a problem in cases where we mark a queue as 'parent' which has no
> configured child queues to start with, but we can add child queues later on.
> Also the exception message is kind of misleading since we haven't configured
> 'reservation'.
> How to reproduce:
> Run fair scheduler with following queue config:
> {code}
>
> 10
> 300
>
> 3
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4975) Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues
[
https://issues.apache.org/jira/browse/YARN-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840606#comment-15840606
]
Yufei Gu commented on YARN-4975:
Thanks [~templedf] for the review and commit!
> Fair Scheduler: exception thrown when a parent queue marked 'parent' has
> configured child queues
>
>
> Key: YARN-4975
> URL: https://issues.apache.org/jira/browse/YARN-4975
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.7.2
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-4975.001.patch, YARN-4975.002.patch
>
>
> We upgraded our clusters to 2.7.2 from 2.4.1 and saw the following exception
> in RM logs :
> {code}
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException:
> Both and type="parent" found for queue root.adhoc which is
> unsupported
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.loadQueue(AllocationFileLoaderService.java:519)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:352)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1440)
> {code}
> From the exception, it looks like we've configured 'reservation', but we've
> not. The issue is that AllocationFileLoaderService#loadQueue assumes that a
> parent queue marked as 'type=parent' cannot have configured child queues.
> That can be a problem in cases where we mark a queue as 'parent' which has no
> configured child queues to start with, but we can add child queues later on.
> Also the exception message is kind of misleading since we haven't configured
> 'reservation'.
> How to reproduce:
> Run fair scheduler with following queue config:
> {code}
>
> 10
> 300
>
> 3
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5258) Document Use of Docker with LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/YARN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5258: --- Attachment: YARN-5258.005.patch Thanks for the reviews, [~shaneku...@gmail.com] and [~sidharta-s]. I have incorporated the feedback. Please have a look at the latest. > Document Use of Docker with LinuxContainerExecutor > -- > > Key: YARN-5258 > URL: https://issues.apache.org/jira/browse/YARN-5258 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.8.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Labels: oct16-easy > Attachments: YARN-5258.001.patch, YARN-5258.002.patch, > YARN-5258.003.patch, YARN-5258.004.patch, YARN-5258.005.patch > > > There aren't currently any docs that explain how to configure Docker and all > of its various options aside from reading all of the JIRAs. We need to > document the configuration, use, and troubleshooting, along with helpful > examples. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6112) fsOpDurations.addUpdateCallDuration() should be independent to LOG level
[
https://issues.apache.org/jira/browse/YARN-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6112:
---
Attachment: YARN-6112.003.patch
> fsOpDurations.addUpdateCallDuration() should be independent to LOG level
>
>
> Key: YARN-6112
> URL: https://issues.apache.org/jira/browse/YARN-6112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6112.001.patch, YARN-6112.002.patch,
> YARN-6112.003.patch
>
>
> In the update thread of Fair Scheduler, the
> {{fsOpDurations.addUpdateCallDuration()}} records the duration of
> {{update()}}, it should be independent to LOG level. YARN-4752 put the it
> inside a {{LOG.isDebugEnabled()}} block. Not sure any particular reason to do
> that. cc [~kasha]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
