[jira] [Comment Edited] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[
https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884985#comment-15884985
]
Rajesh Balamohan edited comment on MAPREDUCE-6850 at 2/27/17 2:15 AM:
--
I checked in small multi-node cluster with the patch. Attaching the tcpdump
screenshots for reference. Patch works fine with keep-alive enabled and
connections are being reused, where mapOutputs are retrieved using same
connection. Attachment "With_Patch.png" shows the TCP stream, where multiple
mapOutput being fetched from same connection.
One very minor comment in the patch. {{timer}} variable in
{{HttpPipelineFactory}} may not be needed.
In MAPREDUCE-5787, Keepalive parameter checks were there till
https://issues.apache.org/jira/secure/attachment/12634984/MAPREDUCE-5787-2.4.0-v3.patch
as follows.
{noformat}
if (!keepAlive && !keepAliveParam) {
lastMap.addListener(ChannelFutureListener.CLOSE);
}
{noformat}
However, during refactoring it got missed out in subsequent patches in the same
JIRA. That caused this problem. However, It would have relied on client to
close the connection. I.e it was the responsibility of the client (JDK's
internal http client) to terminate the connection after keep-alive timeout.
Current patch proposed in this JIRA addresses that scenario as well, where in
it would automatically close the connection if timeout exceeds the threshold
provided in server side.
was (Author: rajesh.balamohan):
I checked in small multi-node cluster with the patch. Attaching the tcpdump
screenshots for reference. Patch works fine with keep-alive enabled and
connections are being reused, where mapOutputs are retrieved using same
connection. Attachment "With_Patch.png" shows the TCP stream, where multiple
mapOutput being fetched from same connection.
One very minor comment in the patch. {{timer}} variable in
{{HttpPipelineFactory}} may not be needed.
In MAPREDUCE-5787, Keepalive parameter checks were there till
https://issues.apache.org/jira/secure/attachment/12634984/MAPREDUCE-5787-2.4.0-v3.patch
as follows.
{noformat}
if (!keepAlive && !keepAliveParam) {
lastMap.addListener(ChannelFutureListener.CLOSE);
}
{noformat}
However, during refactoring it got missed out in subsequent patches. That
caused this problem. However, It would have relied on client to close the
connection. I.e it was the responsibility of the client (JDK's internal http
client) to terminate the connection after keep-alive timeout. Current patch
proposed in this JIRA addresses that scenario as well, where in it would
automatically close the connection if timeout exceeds the threshold provided in
server side.
> Shuffle Handler keep-alive connections are closed from the server side
> --
>
> Key: MAPREDUCE-6850
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch,
> MAPREDUCE-6850.3.patch, With_Issue.png, With_Patch.png,
> With_Patch_withData.png
>
>
> When performance testing tez shuffle handler (TEZ-3334), it was noticed the
> keep-alive connections are closed from the server-side. The client silently
> recovers and logs the connection as keep-alive, despite reestablishing a
> connection. This jira aims to remove the close from the server side, fixing
> the bug preventing keep-alive connections.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[
https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated MAPREDUCE-6850:
Attachment: With_Patch_withData.png
With_Patch.png
With_Issue.png
I checked in small multi-node cluster with the patch. Attaching the tcpdump
screenshots for reference. Patch works fine with keep-alive enabled and
connections are being reused, where mapOutputs are retrieved using same
connection. Attachment "With_Patch.png" shows the TCP stream, where multiple
mapOutput being fetched from same connection.
One very minor comment in the patch. {{timer}} variable in
{{HttpPipelineFactory}} may not be needed.
In MAPREDUCE-5787, Keepalive parameter checks were there till
https://issues.apache.org/jira/secure/attachment/12634984/MAPREDUCE-5787-2.4.0-v3.patch
as follows.
{noformat}
if (!keepAlive && !keepAliveParam) {
lastMap.addListener(ChannelFutureListener.CLOSE);
}
{noformat}
However, during refactoring it got missed out in subsequent patches. That
caused this problem. However, It would have relied on client to close the
connection. I.e it was the responsibility of the client (JDK's internal http
client) to terminate the connection after keep-alive timeout. Current patch
proposed in this JIRA addresses that scenario as well, where in it would
automatically close the connection if timeout exceeds the threshold provided in
server side.
> Shuffle Handler keep-alive connections are closed from the server side
> --
>
> Key: MAPREDUCE-6850
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch,
> MAPREDUCE-6850.3.patch, With_Issue.png, With_Patch.png,
> With_Patch_withData.png
>
>
> When performance testing tez shuffle handler (TEZ-3334), it was noticed the
> keep-alive connections are closed from the server-side. The client silently
> recovers and logs the connection as keep-alive, despite reestablishing a
> connection. This jira aims to remove the close from the server side, fixing
> the bug preventing keep-alive connections.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6753) Variable in byte printed directly in mapreduce client
[
https://issues.apache.org/jira/browse/MAPREDUCE-6753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884791#comment-15884791
]
Hadoop QA commented on MAPREDUCE-6753:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 115m 20s
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 2s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854768/MAPREDUCE-6753.03.patch
|
| JIRA Issue | MAPREDUCE-6753 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fab43a6cc98f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 05391c1 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6905/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6905/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Variable in byte printed directly in mapreduce client
> -
>
> Key: MAPREDUCE-6753
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6753
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.7.2
>Reporter: Nemo Chen
>Assignee: Kai Sasaki
> Labels: easyfix, easytest
> Attachments:
[jira] [Updated] (MAPREDUCE-6753) Variable in byte printed directly in mapreduce client
[
https://issues.apache.org/jira/browse/MAPREDUCE-6753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Sasaki updated MAPREDUCE-6753:
--
Attachment: MAPREDUCE-6753.03.patch
> Variable in byte printed directly in mapreduce client
> -
>
> Key: MAPREDUCE-6753
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6753
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.7.2
>Reporter: Nemo Chen
>Assignee: Kai Sasaki
> Labels: easyfix, easytest
> Attachments: MAPREDUCE-6753.01.patch, MAPREDUCE-6753.02.patch,
> MAPREDUCE-6753.03.patch
>
>
> Similar to the fix for HBASE-623, in file:
> hadoop-rel-release-2.7.2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java
> in line 61, the system out print a byte variable secretValue.
> {code}
> System.out.println(secretValue);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6753) Variable in byte printed directly in mapreduce client
[
https://issues.apache.org/jira/browse/MAPREDUCE-6753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15884748#comment-15884748
]
Kai Sasaki commented on MAPREDUCE-6753:
---
[~haibochen] Thanks for comment. I think the both case. Secret value is thought
not to be printed initially. But anyway it's printed as byte[] value. We cannot
see the secret value even if it's printed. So it should be printed as string if
it needs to be printed.
> Variable in byte printed directly in mapreduce client
> -
>
> Key: MAPREDUCE-6753
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6753
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.7.2
>Reporter: Nemo Chen
>Assignee: Kai Sasaki
> Labels: easyfix, easytest
> Attachments: MAPREDUCE-6753.01.patch, MAPREDUCE-6753.02.patch
>
>
> Similar to the fix for HBASE-623, in file:
> hadoop-rel-release-2.7.2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java
> in line 61, the system out print a byte variable secretValue.
> {code}
> System.out.println(secretValue);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
