[jira] [Commented] (YARN-7668) Remove unused variables from ContainerLocalizer
[
https://issues.apache.org/jira/browse/YARN-7668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516674#comment-16516674
]
ASF GitHub Bot commented on YARN-7668:
--
Github user dedunumax commented on the issue:
https://github.com/apache/hadoop/pull/364
dedunumax is my JIRA ID.
> Remove unused variables from ContainerLocalizer
> ---
>
> Key: YARN-7668
> URL: https://issues.apache.org/jira/browse/YARN-7668
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Ray Chiang
>Assignee: Dedunu Dhananjaya
>Priority: Trivial
> Labels: newbie
> Fix For: 3.2.0
>
> Attachments: 364.patch
>
>
> While figuring out something else, I found two class constants in
> ContainerLocalizer that look like aren't being used anymore.
> {noformat}
> public static final String OUTPUTDIR = "output";
> public static final String WORKDIR = "work";
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8234) Improve RM system metrics publisher's performance by pushing events to timeline server in batch
[ https://issues.apache.org/jira/browse/YARN-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516585#comment-16516585 ] Hu Ziqian commented on YARN-8234: - Hi [~leftnoteasy]，i‘ve added another patch YARN-8234.003.patch which implements this issue on trunk. I've fix all unit test and the findbugs warning is not imported by this patch. The only problem left is shadedclient. I don't understand what is this error means. Can you tell me how to replay it on my laptop environment. Do i need to add some flags when compiling by maven? Thank you > Improve RM system metrics publisher's performance by pushing events to > timeline server in batch > --- > > Key: YARN-8234 > URL: https://issues.apache.org/jira/browse/YARN-8234 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, timelineserver >Affects Versions: 2.8.3 >Reporter: Hu Ziqian >Assignee: Hu Ziqian >Priority: Critical > Attachments: YARN-8234-branch-2.8.3.001.patch, > YARN-8234-branch-2.8.3.002.patch, YARN-8234-branch-2.8.3.003.patch, > YARN-8234.001.patch, YARN-8234.002.patch, YARN-8234.003.patch > > > When system metrics publisher is enabled, RM will push events to timeline > server via restful api. If the cluster load is heavy, many events are sent to > timeline server and the timeline server's event handler thread locked. > YARN-7266 talked about the detail of this problem. Because of the lock, > timeline server can't receive event as fast as it generated in RM and lots of > timeline event stays in RM's memory. Finally, those events will consume all > RM's memory and RM will start a full gc (which cause an JVM stop-world and > cause a timeout from rm to zookeeper) or even get an OOM. > The main problem here is that timeline can't receive timeline server's event > as fast as it generated. Now, RM system metrics publisher put only one event > in a request, and most time costs on handling http header or some thing about > the net connection on timeline side. Only few time is spent on dealing with > the timeline event which is truly valuable. > In this issue, we add a buffer in system metrics publisher and let publisher > send events to timeline server in batch via one request. When sets the batch > size to 1000, in out experiment the speed of the timeline server receives > events has 100x improvement. We have implement this function int our product > environment which accepts 2 app's in one hour and it works fine. > We add following configuration: > * yarn.resourcemanager.system-metrics-publisher.batch-size: the size of > system metrics publisher sending events in one request. Default value is 1000 > * yarn.resourcemanager.system-metrics-publisher.buffer-size: the size of the > event buffer in system metrics publisher. > * yarn.resourcemanager.system-metrics-publisher.interval-seconds: When > enable batch publishing, we must avoid that the publisher waits for a batch > to be filled up and hold events in buffer for long time. So we add another > thread which send event's in the buffer periodically. This config sets the > interval of the cyclical sending thread. The default value is 60s. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7712) Add ability to ignore timestamps in localized files
[ https://issues.apache.org/jira/browse/YARN-7712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi reassigned YARN-7712: Assignee: (was: Miklos Szegedi) > Add ability to ignore timestamps in localized files > --- > > Key: YARN-7712 > URL: https://issues.apache.org/jira/browse/YARN-7712 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Miklos Szegedi >Priority: Major > > YARN currently requires and checks the timestamp of localized files and > fails, if the file on HDFS does not match to the one requested. This jira > adds the ability to ignore the timestamp based on the request of the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7712) Add ability to ignore timestamps in localized files
[ https://issues.apache.org/jira/browse/YARN-7712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516509#comment-16516509 ] Miklos Szegedi commented on YARN-7712: -- Since I really needed this only for testing, let's defer this for now. > Add ability to ignore timestamps in localized files > --- > > Key: YARN-7712 > URL: https://issues.apache.org/jira/browse/YARN-7712 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Major > > YARN currently requires and checks the timestamp of localized files and > fails, if the file on HDFS does not match to the one requested. This jira > adds the ability to ignore the timestamp based on the request of the client. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7713) Add parallel copying of directories into FSDownload
[ https://issues.apache.org/jira/browse/YARN-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-7713: - Labels: newbie (was: ) > Add parallel copying of directories into FSDownload > --- > > Key: YARN-7713 > URL: https://issues.apache.org/jira/browse/YARN-7713 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Major > Labels: newbie > > YARN currently copies directories sequentially when localizing. This could be > improved to do in parallel, since the source blocks are normally on different > nodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7713) Add parallel copying of directories into FSDownload
[ https://issues.apache.org/jira/browse/YARN-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi reassigned YARN-7713: Assignee: (was: Miklos Szegedi) > Add parallel copying of directories into FSDownload > --- > > Key: YARN-7713 > URL: https://issues.apache.org/jira/browse/YARN-7713 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Miklos Szegedi >Priority: Major > Labels: newbie > > YARN currently copies directories sequentially when localizing. This could be > improved to do in parallel, since the source blocks are normally on different > nodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5986) Add support for container specific properties to the ContainerLaunchContext
[
https://issues.apache.org/jira/browse/YARN-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi reassigned YARN-5986:
Assignee: (was: Miklos Szegedi)
> Add support for container specific properties to the ContainerLaunchContext
> ---
>
> Key: YARN-5986
> URL: https://issues.apache.org/jira/browse/YARN-5986
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Miklos Szegedi
>Priority: Major
>
> Currently ContainerLaunchContext is defined as
> message ContainerLaunchContextProto {
> repeated StringLocalResourceMapProto localResources = 1;
> optional bytes tokens = 2;
> repeated StringBytesMapProto service_data = 3;
> repeated StringStringMapProto environment = 4;
> repeated string command = 5;
> repeated ApplicationACLMapProto application_ACLs = 6;
> optional ContainerRetryContextProto container_retry_context = 7;
> }
> It would be nice to have an additional parameter "configuration" to support
> cases like YARN-5600, where we want to pass a parameter to Yarn and not the
> application or container.
> repeated StringStringMapProto configuration = 8;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6794) Fair Scheduler to explicitly promote OPPORTUNISITIC containers locally at the node where they're running
[
https://issues.apache.org/jira/browse/YARN-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516505#comment-16516505
]
Miklos Szegedi commented on YARN-6794:
--
Thank you for the patch [~haibochen]. I have the following comments:
{code:java}
283 Resources.subtractFrom(allocatedResourceGuaranteed, resource);{code}
I think this should refer to allocatedResourceOpportunistic
{code:java}
526 Resource resource = rmContainer.getContainer().getResource();
527 if (Resources.fitsIn(resource, node.getUnallocatedResource())) {
528 node.promoteOpportunisticContainer(rmContainer);{code}
This is outside the critical section. The unallocated space may change after
the check but before the promotion.
{code:java}
529 attemptOpportunisticResourceUsage.decUsed(resource);
530 attemptResourceUsage.incUsed(resource);
531 getQueue().incUsedGuaranteedResource(resource);{code}
SchedulerApplicationAttempt.getResourceUsageReport may return a transient 0
since the write lock is not held at this point.
{code:java}
1391public Resource getGuaranteedResourceUsage() {{code}
It might be safer to do a clone here and return a read only copy.
{code:java}
162 @Override
163 public void promoteOpportunisticContainer(RMContainer rmContainer) {
164 super.promoteOpportunisticContainer(rmContainer);
165 priorityContainers.remove(rmContainer);
166 }{code}
This might cause a double promotion. The function is not synchronized, so we do
the promotion but priorityContainers can be sampled again with the same
container listed before this finishes.
{code:java}
321 public List getPriorityContainers() {{code}
It is probably a good idea to have this synchronized as well. The list can
change while you grab the snapshot.
{code:java}
1124//boolean validReservation =
attemptToAssignReservedResources(node);{code}
?
{code:java}
1165if (!assigned) {
1166// break out of the loop because assigned being false
1167// indicates there is no more resources are available.
1168break;
1169}{code}
Is this really true? I could imagine that we try but cannot assign a reserved
resource but we do not even try the opportunistic queue in that case.
> Fair Scheduler to explicitly promote OPPORTUNISITIC containers locally at the
> node where they're running
>
>
> Key: YARN-6794
> URL: https://issues.apache.org/jira/browse/YARN-6794
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, scheduler
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-6794-YARN-1011.00.patch,
> YARN-6794-YARN-1011.prelim.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7668) Remove unused variables from ContainerLocalizer
[
https://issues.apache.org/jira/browse/YARN-7668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516481#comment-16516481
]
Hudson commented on YARN-7668:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14449 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14449/])
YARN-7668. Remove unused variables from ContainerLocalizer (aajisaka: rev
f386e78a4bc74e7c247b179c7d4ec27310fda4d3)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
> Remove unused variables from ContainerLocalizer
> ---
>
> Key: YARN-7668
> URL: https://issues.apache.org/jira/browse/YARN-7668
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Ray Chiang
>Assignee: Dedunu Dhananjaya
>Priority: Trivial
> Labels: newbie
> Fix For: 3.2.0
>
> Attachments: 364.patch
>
>
> While figuring out something else, I found two class constants in
> ContainerLocalizer that look like aren't being used anymore.
> {noformat}
> public static final String OUTPUTDIR = "output";
> public static final String WORKDIR = "work";
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7668) Remove unused variables from ContainerLocalizer
[
https://issues.apache.org/jira/browse/YARN-7668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516473#comment-16516473
]
ASF GitHub Bot commented on YARN-7668:
--
Github user asfgit closed the pull request at:
https://github.com/apache/hadoop/pull/364
> Remove unused variables from ContainerLocalizer
> ---
>
> Key: YARN-7668
> URL: https://issues.apache.org/jira/browse/YARN-7668
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Ray Chiang
>Assignee: Dedunu Dhananjaya
>Priority: Trivial
> Labels: newbie
> Fix For: 3.2.0
>
> Attachments: 364.patch
>
>
> While figuring out something else, I found two class constants in
> ContainerLocalizer that look like aren't being used anymore.
> {noformat}
> public static final String OUTPUTDIR = "output";
> public static final String WORKDIR = "work";
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7668) Remove unused variables from ContainerLocalizer
[
https://issues.apache.org/jira/browse/YARN-7668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reassigned YARN-7668:
---
Assignee: Dedunu Dhananjaya
> Remove unused variables from ContainerLocalizer
> ---
>
> Key: YARN-7668
> URL: https://issues.apache.org/jira/browse/YARN-7668
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Ray Chiang
>Assignee: Dedunu Dhananjaya
>Priority: Trivial
> Labels: newbie
> Attachments: 364.patch
>
>
> While figuring out something else, I found two class constants in
> ContainerLocalizer that look like aren't being used anymore.
> {noformat}
> public static final String OUTPUTDIR = "output";
> public static final String WORKDIR = "work";
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7449) Split up class TestYarnClient to TestYarnClient and TestYarnClientImpl
[
https://issues.apache.org/jira/browse/YARN-7449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516449#comment-16516449
]
genericqa commented on YARN-7449:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
37s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 1m
59s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 8s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMProxy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | YARN-7449 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928263/YARN-7449-001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux ea5233701ef9 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e94e59 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/21051/artifact/out/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/21051/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21051/testReport/ |
| Max. process+thread count | 707 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoo
[jira] [Commented] (YARN-7668) Remove unused variables from ContainerLocalizer
[
https://issues.apache.org/jira/browse/YARN-7668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516442#comment-16516442
]
genericqa commented on YARN-7668:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
33s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 1m
40s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
33s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | YARN-7668 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928264/364.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 41a9a8d09ab5 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e94e59 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21050/testReport/ |
| Max. process+thread count | 180 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21050/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
[jira] [Updated] (YARN-7668) Remove unused variables from ContainerLocalizer
[
https://issues.apache.org/jira/browse/YARN-7668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-7668:
Attachment: 364.patch
> Remove unused variables from ContainerLocalizer
> ---
>
> Key: YARN-7668
> URL: https://issues.apache.org/jira/browse/YARN-7668
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Ray Chiang
>Priority: Trivial
> Labels: newbie
> Attachments: 364.patch
>
>
> While figuring out something else, I found two class constants in
> ContainerLocalizer that look like aren't being used anymore.
> {noformat}
> public static final String OUTPUTDIR = "output";
> public static final String WORKDIR = "work";
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7668) Remove unused variables from ContainerLocalizer
[
https://issues.apache.org/jira/browse/YARN-7668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516379#comment-16516379
]
ASF GitHub Bot commented on YARN-7668:
--
Github user aajisaka commented on the issue:
https://github.com/apache/hadoop/pull/364
LGTM, +1.
@dedunumax Would you please tell me your JIRA ID? I'll assign you to the
issue.
> Remove unused variables from ContainerLocalizer
> ---
>
> Key: YARN-7668
> URL: https://issues.apache.org/jira/browse/YARN-7668
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Ray Chiang
>Priority: Trivial
> Labels: newbie
>
> While figuring out something else, I found two class constants in
> ContainerLocalizer that look like aren't being used anymore.
> {noformat}
> public static final String OUTPUTDIR = "output";
> public static final String WORKDIR = "work";
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7449) Split up class TestYarnClient to TestYarnClient and TestYarnClientImpl
[
https://issues.apache.org/jira/browse/YARN-7449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-7449:
-
Attachment: YARN-7449-001.patch
> Split up class TestYarnClient to TestYarnClient and TestYarnClientImpl
> --
>
> Key: YARN-7449
> URL: https://issues.apache.org/jira/browse/YARN-7449
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, yarn
>Reporter: Yufei Gu
>Assignee: Szilard Nemeth
>Priority: Minor
> Labels: newbie, newbie++
> Attachments: YARN-7449-001.patch
>
>
> {{TestYarnClient}} tests both {{YarnClient}} and {{YarnClientImpl}}. We
> should test {{YarnClient}} without thinking of its implementation. That's the
> whole point of {{YarnClient}}. There are bunch of refactors we could do. The
> first thing is to Split up class {{TestYarnClient}} to {{TestYarnClient}} and
> {{TestYarnClientImpl}}. Let {{TestYarnClient}} only tests {{YarnClient}}. All
> implementation related stuff go to {{TestYarnClientImpl}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8391) Investigate AllocationFileLoaderService.reloadListener locking issue
[
https://issues.apache.org/jira/browse/YARN-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516316#comment-16516316
]
genericqa commented on YARN-8391:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
41s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
10s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 1m
51s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
16s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m
11s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | Inconsistent synchronization of
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.fs;
locked 50% of time Unsynchronized access at
AllocationFileLoaderService.java:50% of time Unsynchronized access at
AllocationFileLoaderService.java:[line 119] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | YARN-8391 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928241/YARN-8191.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 025a2ed60765 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /test
[jira] [Commented] (YARN-8320) [Umbrella] Support CPU isolation for latency-sensitive (LS) service
[ https://issues.apache.org/jira/browse/YARN-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516290#comment-16516290 ] Miklos Szegedi commented on YARN-8320: -- [~cheersyang], thank you for the response. Indeed, I was assuming users will set some typical values for vcores(cores) that translate to cpu.shares and cpuset. These are like 0.2, 0.5 for shareable and 1.0 or 2.0 for exclusive. Values like 4.5 are too hard to understand indeed. Please go ahead with your approach. It would be useful to keep in mind that the system needs to be able to set both cpu.shares and cpuset dimensions some time in the future. > [Umbrella] Support CPU isolation for latency-sensitive (LS) service > --- > > Key: YARN-8320 > URL: https://issues.apache.org/jira/browse/YARN-8320 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Reporter: Jiandan Yang >Priority: Major > Attachments: CPU-isolation-for-latency-sensitive-services-v1.pdf, > CPU-isolation-for-latency-sensitive-services-v2.pdf, YARN-8320.001.patch > > > Currently NodeManager uses “cpu.cfs_period_us”, “cpu.cfs_quota_us” and > “cpu.shares” to isolate cpu resource. However, > * Linux Completely Fair Scheduling (CFS) is a throughput-oriented scheduler; > no support for differentiated latency > * Request latency of services running on container may be frequent shake > when all containers share cpus, and latency-sensitive services can not afford > in our production environment. > So we need more fine-grained cpu isolation. > Here we propose a solution using cgroup cpuset to binds containers to > different processors, this is inspired by the isolation technique in [Borg > system|http://schd.ws/hosted_files/lcccna2016/a7/CAT%20@%20Scale.pdf]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8391) Investigate AllocationFileLoaderService.reloadListener locking issue
[ https://issues.apache.org/jira/browse/YARN-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8391: - Attachment: YARN-8191.001.patch > Investigate AllocationFileLoaderService.reloadListener locking issue > > > Key: YARN-8391 > URL: https://issues.apache.org/jira/browse/YARN-8391 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.2.0 >Reporter: Haibo Chen >Assignee: Szilard Nemeth >Priority: Critical > Attachments: YARN-8191.001.patch > > > Per findbugs report in YARN-8390, there is some inconsistent locking of > reloadListener > > h1. Warnings > Click on a warning row to see full context information. > h2. Multithreaded correctness Warnings > ||Code||Warning|| > |IS|Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadListener; > locked 75% of time| > | |[Bug type IS2_INCONSISTENT_SYNC (click for > details)|https://builds.apache.org/job/PreCommit-YARN-Build/20939/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html#IS2_INCONSISTENT_SYNC] > > In class > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService > Field > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadListener > Synchronized 75% of the time > Unsynchronized access at AllocationFileLoaderService.java:[line 117] > Synchronized access at AllocationFileLoaderService.java:[line 212] > Synchronized access at AllocationFileLoaderService.java:[line 228] > Synchronized access at AllocationFileLoaderService.java:[line 269]| > h1. Details > h2. IS2_INCONSISTENT_SYNC: Inconsistent synchronization > The fields of this class appear to be accessed inconsistently with respect to > synchronization. This bug report indicates that the bug pattern detector > judged that > * The class contains a mix of locked and unlocked accesses, > * The class is *not* annotated as javax.annotation.concurrent.NotThreadSafe, > * At least one locked access was performed by one of the class's own > methods, and > * The number of unsynchronized field accesses (reads and writes) was no more > than one third of all accesses, with writes being weighed twice as high as > reads > A typical bug matching this bug pattern is forgetting to synchronize one of > the methods in a class that is intended to be thread-safe. > You can select the nodes labeled "Unsynchronized access" to show the code > locations where the detector believed that a field was accessed without > synchronization. > Note that there are various sources of inaccuracy in this detector; for > example, the detector cannot statically detect all situations in which a lock > is held. Also, even when the detector is accurate in distinguishing locked > vs. unlocked accesses, the code in question may still be correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8103) Add CLI interface to query node attributes
[ https://issues.apache.org/jira/browse/YARN-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516100#comment-16516100 ] Naganarasimha G R commented on YARN-8103: - Thanks for the patch [~bibinchundatt] Some of my comments seems to be still not addressed: * NodeAttributesCLI ln no 187: i think its better to use null here instead of "handler" for better readability * TestNodeAttributesCLI ln no 405: testListAttributes is encapsulating NodesToAttributes tests too, may be it can captured as a different case * NodeCLI ln no 317 : IMO it would better to wrap each attribute in a new line ? And some of the [~cheersyang] comments needs to be discussed (though i am not more towards numeric form for to but we can discuss other better options) and this comment needs to be handled * line 588: buildNodeLabelsMapFromStr -> buildNodeAttributesListFromStr Apart from this many of the Checkstyle issues still seems to be valid and needs to be addressed. > Add CLI interface to query node attributes > --- > > Key: YARN-8103 > URL: https://issues.apache.org/jira/browse/YARN-8103 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-8103-YARN-3409.001.patch, > YARN-8103-YARN-3409.002.patch, YARN-8103-YARN-3409.003.patch, > YARN-8103-YARN-3409.WIP.patch > > > YARN-8100 will add API interface for querying the attributes. CLI interface > for querying node attributes for each nodes and list all attributes in > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8326) Yarn 3.0 seems runs slower than Yarn 2.6
[ https://issues.apache.org/jira/browse/YARN-8326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516072#comment-16516072 ] Eric Yang commented on YARN-8326: - [~shaneku...@gmail.com] In line [802|https://github.com/apache/hadoop/blob/fba9d7cd746cd7b659d2fd9d2bfa23266be9009b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L804] of ContainerLaunch.java, the exit_code file is deleted. By the time that YARN-5366 sleeping loop is encountered, the file does not exist. This is the reason that 2 second sleep always happens. I mentioned in YARN-5366, I did not find sleep and retry docker inspect logic in YARN-5366, and I added in YARN-7654 line [1671|https://github.com/apache/hadoop/blame/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c#L1671] to have the retry logic for entry_point based docker container. I think the two seconds sleep logic can be removed from ContainerLaunch.java. Thoughts? > Yarn 3.0 seems runs slower than Yarn 2.6 > > > Key: YARN-8326 > URL: https://issues.apache.org/jira/browse/YARN-8326 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0 > Environment: This is the yarn-site.xml for 3.0. > > > > hadoop.registry.dns.bind-port > 5353 > > > hadoop.registry.dns.domain-name > hwx.site > > > hadoop.registry.dns.enabled > true > > > hadoop.registry.dns.zone-mask > 255.255.255.0 > > > hadoop.registry.dns.zone-subnet > 172.17.0.0 > > > manage.include.files > false > > > yarn.acl.enable > false > > > yarn.admin.acl > yarn > > > yarn.client.nodemanager-connect.max-wait-ms > 6 > > > yarn.client.nodemanager-connect.retry-interval-ms > 1 > > > yarn.http.policy > HTTP_ONLY > > > yarn.log-aggregation-enable > false > > > yarn.log-aggregation.retain-seconds > 2592000 > > > yarn.log.server.url > > [http://xx:19888/jobhistory/logs|http://whiny2.fyre.ibm.com:19888/jobhistory/logs] > > > yarn.log.server.web-service.url > > [http://xx:8188/ws/v1/applicationhistory|http://whiny2.fyre.ibm.com:8188/ws/v1/applicationhistory] > > > yarn.node-labels.enabled > false > > > yarn.node-labels.fs-store.retry-policy-spec > 2000, 500 > > > yarn.node-labels.fs-store.root-dir > /system/yarn/node-labels > > > yarn.nodemanager.address > 0.0.0.0:45454 > > > yarn.nodemanager.admin-env > MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX > > > yarn.nodemanager.aux-services > mapreduce_shuffle,spark2_shuffle,timeline_collector > > > yarn.nodemanager.aux-services.mapreduce_shuffle.class > org.apache.hadoop.mapred.ShuffleHandler > > > yarn.nodemanager.aux-services.spark2_shuffle.class > org.apache.spark.network.yarn.YarnShuffleService > > > yarn.nodemanager.aux-services.spark2_shuffle.classpath > /usr/spark2/aux/* > > > yarn.nodemanager.aux-services.spark_shuffle.class > org.apache.spark.network.yarn.YarnShuffleService > > > yarn.nodemanager.aux-services.timeline_collector.class > > org.apache.hadoop.yarn.server.timelineservice.collector.PerNodeTimelineCollectorsAuxService > > > yarn.nodemanager.bind-host > 0.0.0.0 > > > yarn.nodemanager.container-executor.class > > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor > > > yarn.nodemanager.container-metrics.unregister-delay-ms > 6 > > > yarn.nodemanager.container-monitor.interval-ms > 3000 > > > yarn.nodemanager.delete.debug-delay-sec > 0 > > > > yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage > 90 > > > yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb > 1000 > > > yarn.nodemanager.disk-health-checker.min-healthy-disks > 0.25 > > > yarn.nodemanager.health-checker.interval-ms > 135000 > > > yarn.nodemanager.health-checker.script.timeout-ms > 6 > > > > yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage > false > > > yarn.nodemanager.linux-container-executor.group > hadoop > > > > yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users > false > > > yarn.nodemanager.local-dirs > /hadoop/yarn/local > > > yarn.nodemanager.log-aggregation.compression-type > gz > > > yarn.nodemanager.log-aggregation.debug-enabled > false > > > yarn.nodemanager.log-aggregation.num-log-files-per-app > 30 > > > > yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds > 3600 > > > yarn.nodemanager.log-dirs > /hadoop/yarn/log > > > yarn.nodemanager.log.retain-seconds > 604800 > > > yarn.nodeman
[jira] [Commented] (YARN-8412) Move ResourceRequest.clone logic everywhere into a proper API
[ https://issues.apache.org/jira/browse/YARN-8412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516061#comment-16516061 ] Íñigo Goiri commented on YARN-8412: --- Thanks [~botong] for the explanation. TestAppManager is the only weird plain and it's a unit test that passes so we should be fine there. +1 Where do you want to commit this? > Move ResourceRequest.clone logic everywhere into a proper API > - > > Key: YARN-8412 > URL: https://issues.apache.org/jira/browse/YARN-8412 > Project: Hadoop YARN > Issue Type: Task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-8412-branch-2.v2.patch, YARN-8412.v1.patch, > YARN-8412.v2.patch > > > ResourceRequest.clone code is replicated in lots of places, some missing to > copy one field or two due to new fields added over time. This JIRA attempts > to move them into a proper API so that everyone can use this single > implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8412) Move ResourceRequest.clone logic everywhere into a proper API
[ https://issues.apache.org/jira/browse/YARN-8412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-8412: --- Attachment: YARN-8412-branch-2.v2.patch > Move ResourceRequest.clone logic everywhere into a proper API > - > > Key: YARN-8412 > URL: https://issues.apache.org/jira/browse/YARN-8412 > Project: Hadoop YARN > Issue Type: Task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-8412-branch-2.v2.patch, YARN-8412.v1.patch, > YARN-8412.v2.patch > > > ResourceRequest.clone code is replicated in lots of places, some missing to > copy one field or two due to new fields added over time. This JIRA attempts > to move them into a proper API so that everyone can use this single > implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8412) Move ResourceRequest.clone logic everywhere into a proper API
[ https://issues.apache.org/jira/browse/YARN-8412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-8412: --- Attachment: (was: YARN-8412-branch-2.v2.patch) > Move ResourceRequest.clone logic everywhere into a proper API > - > > Key: YARN-8412 > URL: https://issues.apache.org/jira/browse/YARN-8412 > Project: Hadoop YARN > Issue Type: Task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-8412-branch-2.v2.patch, YARN-8412.v1.patch, > YARN-8412.v2.patch > > > ResourceRequest.clone code is replicated in lots of places, some missing to > copy one field or two due to new fields added over time. This JIRA attempts > to move them into a proper API so that everyone can use this single > implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8234) Improve RM system metrics publisher's performance by pushing events to timeline server in batch
[
https://issues.apache.org/jira/browse/YARN-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515501#comment-16515501
]
genericqa commented on YARN-8234:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 5m
5s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
13s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
31s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
44s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
21s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 68m
24s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | YARN-8234 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928150/YARN-8234.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux dbeb369a7728 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x8
