[jira] [Commented] (MAPREDUCE-6638) Do not attempt to recover progress from previous job attempts if spill encryption is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542955#comment-15542955 ] Hudson commented on MAPREDUCE-6638: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10532 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10532/]) MAPREDUCE-6638. Do not attempt to recover progress from previous job (kasha: rev de7a0a92ca1983b35ca4beb7ab712fd700a9e6e0) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java > Do not attempt to recover progress from previous job attempts if spill > encryption is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6638) Do not attempt to recover progress from previous job attempts if spill encryption is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-6638: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.9.0 Status: Resolved (was: Patch Available) Just committed to trunk and branch-2. Thanks [~haibochen] for cleaning this up, and [~hitesh] for chiming in on the appropriateness of the title/commit message. > Do not attempt to recover progress from previous job attempts if spill > encryption is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[ https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542927#comment-15542927 ] Daniel Templeton commented on MAPREDUCE-6728: - Thanks, [~haibochen]! The patch looks great. Two tiny nits: # The colon in {{ShuffleHandler}} needs a space before it: {code} for (Map.Entry header: headers.entrySet()) {{code} # It would be nice to add a comment where you send the error code to say what the header means and why you're sending it. > Give fetchers hint when ShuffleHandler rejects a shuffling connection > - > > Key: MAPREDUCE-6728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6728.001.patch, mapreduce6728.002.patch, > mapreduce6728.003.patch, mapreduce6728.prelim.patch > > > If # of open shuffle connection to a node goes over the max, ShuffleHandler > closes the connection immediately without giving fetchers any hint of the > reason, which causes fetchers to fail due to exceptions > java.net.SocketException: Unexpected end of file from server > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) > OR > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java > Such failures are counted as fetcher failures -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6638) Do not attempt to recover progress from previous job attempts if spill encryption is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542926#comment-15542926 ] Karthik Kambatla commented on MAPREDUCE-6638: - Thanks Hitesh. Checking this in with the updated title. > Do not attempt to recover progress from previous job attempts if spill > encryption is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6638) Do not attempt to recover progress from previous job attempts if spill encryption is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6638: -- Summary: Do not attempt to recover progress from previous job attempts if spill encryption is enabled (was: Do not attempt to recover jobs if encrypted spill is enabled) > Do not attempt to recover progress from previous job attempts if spill > encryption is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6638) Do not attempt to recover jobs if encrypted spill is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542845#comment-15542845 ] Hitesh Shah edited comment on MAPREDUCE-6638 at 10/3/16 4:51 PM: - Havent looked at the patch in detail but [~haibochen]'s clarifying comments make sense. Jira title could be modified accordingly. +0 from my side. was (Author: hitesh): Havent looked at the patch in detail but [~haibochen]'s clarifying comments make sense. +0 from my side. > Do not attempt to recover jobs if encrypted spill is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6638) Do not attempt to recover jobs if encrypted spill is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542845#comment-15542845 ] Hitesh Shah commented on MAPREDUCE-6638: Havent looked at the patch in detail but [~haibochen]'s clarifying comments make sense. +0 from my side. > Do not attempt to recover jobs if encrypted spill is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6638) Do not attempt to recover jobs if encrypted spill is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542462#comment-15542462 ] Karthik Kambatla commented on MAPREDUCE-6638: - [~hitesh] - are you comfortable with checking this in? > Do not attempt to recover jobs if encrypted spill is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org