[jira] [Commented] (MAPREDUCE-7252) Handling 0 progress in SimpleExponential task runtime estimator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010848#comment-17010848 ] Hudson commented on MAPREDUCE-7252: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17832 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17832/]) MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime (jeagles: rev cdd6efd3ab6917e30b4c5c7b261f61838901bb37) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/forecast/SimpleExponentialSmoothing.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java * (add) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/forecast/package-info.java > Handling 0 progress in SimpleExponential task runtime estimator > --- > > Key: MAPREDUCE-7252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7252 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7252.001.patch, MAPREDUCE-7252.002.patch, > MAPREDUCE-7252.003.patch > > > The simple exponential runtime estimator (added in MAPREDUCE-7208) seems not > to handle the corner cases where the delta progress is 0. As a result, the > forecast will be NaN or Inf which messes up the subsequent forecast values. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7079) TestMRIntermediateDataEncryption is failing in precommit builds
[
https://issues.apache.org/jira/browse/MAPREDUCE-7079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7079:
-
Attachment: MAPREDUCE-7079.003.patch
> TestMRIntermediateDataEncryption is failing in precommit builds
> ---
>
> Key: MAPREDUCE-7079
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7079
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jason Darrell Lowe
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7079.001.patch, MAPREDUCE-7079.002.patch,
> MAPREDUCE-7079.003.patch
>
>
> TestMRIntermediateDataEncryption is either timing out or tearing down the JVM
> which causes the unit tests in jobclient to not pass cleanly during precommit
> builds. From sample precommit console output, note the lack of a test results
> line when the test is run:
> {noformat}
> [INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976
> s - in org.apache.hadoop.mapred.TestSequenceFileInputFormat
> [INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
> [INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659
> s - in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
> [...]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:14 h
> [INFO] Finished at: 2018-04-12T04:27:06+00:00
> [INFO] Final Memory: 24M/594M
> [INFO]
>
> [WARNING] The requested profile "parallel-tests" could not be activated
> because it does not exist.
> [WARNING] The requested profile "native" could not be activated because it
> does not exist.
> [WARNING] The requested profile "yarn-ui" could not be activated because it
> does not exist.
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on
> project hadoop-mapreduce-client-jobclient: There was a timeout or other error
> in the fork -> [Help 1]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7252) Handling 0 progress in SimpleExponential task runtime estimator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles updated MAPREDUCE-7252: -- Fix Version/s: 3.2.2 3.1.4 3.3.0 > Handling 0 progress in SimpleExponential task runtime estimator > --- > > Key: MAPREDUCE-7252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7252 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: MAPREDUCE-7252.001.patch, MAPREDUCE-7252.002.patch, > MAPREDUCE-7252.003.patch > > > The simple exponential runtime estimator (added in MAPREDUCE-7208) seems not > to handle the corner cases where the delta progress is 0. As a result, the > forecast will be NaN or Inf which messes up the subsequent forecast values. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7252) Handling 0 progress in SimpleExponential task runtime estimator
[
https://issues.apache.org/jira/browse/MAPREDUCE-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010854#comment-17010854
]
Jonathan Turner Eagles commented on MAPREDUCE-7252:
---
[~ahussein], could you provide a specific patch for branch-2.10? Getting this
error
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hadoop-mapreduce-client-app: Compilation
failure
[ERROR]
/Users/jeagles/hadoop/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java:[397,61]
method references are not supported in -source 1.7
[ERROR] (use -source 8 or higher to enable method references)
{noformat}
Which corresponds to this code.
{code}
String statesValues = targetStates.stream().map(Object::toString).collect(
Collectors.joining(","));
{code}
> Handling 0 progress in SimpleExponential task runtime estimator
> ---
>
> Key: MAPREDUCE-7252
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7252
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: MAPREDUCE-7252.001.patch, MAPREDUCE-7252.002.patch,
> MAPREDUCE-7252.003.patch
>
>
> The simple exponential runtime estimator (added in MAPREDUCE-7208) seems not
> to handle the corner cases where the delta progress is 0. As a result, the
> forecast will be NaN or Inf which messes up the subsequent forecast values.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7252) Handling 0 progress in SimpleExponential task runtime estimator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7252: - Attachment: MAPREDUCE-7252-branch-2.10.003.patch > Handling 0 progress in SimpleExponential task runtime estimator > --- > > Key: MAPREDUCE-7252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7252 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: MAPREDUCE-7252-branch-2.10.003.patch, > MAPREDUCE-7252.001.patch, MAPREDUCE-7252.002.patch, MAPREDUCE-7252.003.patch > > > The simple exponential runtime estimator (added in MAPREDUCE-7208) seems not > to handle the corner cases where the delta progress is 0. As a result, the > forecast will be NaN or Inf which messes up the subsequent forecast values. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7079) TestMRIntermediateDataEncryption is failing in precommit builds
[
https://issues.apache.org/jira/browse/MAPREDUCE-7079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011031#comment-17011031
]
Hadoop QA commented on MAPREDUCE-7079:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client:
The patch generated 0 new + 9 unchanged - 24 fixed = 9 total (was 33) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
32s{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}151m 51s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
41s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}234m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.TestMRJobClient |
| | hadoop.mapreduce.v2.TestUberAM |
| | hadoop.mapreduce.security.ssl.TestEncryptedShuffle |
| | hadoop.mapreduce.v2.TestMRJobs |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | MAPREDUCE-7079 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12990231/MAPREDUCE-7079.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 32aab82bfcc4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Per
[jira] [Commented] (MAPREDUCE-7252) Handling 0 progress in SimpleExponential task runtime estimator
[
https://issues.apache.org/jira/browse/MAPREDUCE-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011050#comment-17011050
]
Hadoop QA commented on MAPREDUCE-7252:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.10 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
11s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} branch-2.10 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} branch-2.10 passed with JDK v1.8.0_232 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} branch-2.10 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} branch-2.10 passed with JDK v1.8.0_232 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed with JDK v1.8.0_232 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 2 new +
64 unchanged - 8 fixed = 66 total (was 72) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.8.0_232 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
16s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 23s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
42s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}184m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:a969cad0a12 |
| JIRA Issue | MAPREDUCE-7252 |
| JIRA Patch URL |
http
