[jira] [Commented] (MAPREDUCE-7252) Handling 0 progress in SimpleExponential task runtime estimator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010848#comment-17010848 ] Hudson commented on MAPREDUCE-7252: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17832 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17832/]) MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime (jeagles: rev cdd6efd3ab6917e30b4c5c7b261f61838901bb37) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/forecast/SimpleExponentialSmoothing.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java * (add) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/forecast/package-info.java > Handling 0 progress in SimpleExponential task runtime estimator > --- > > Key: MAPREDUCE-7252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7252 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7252.001.patch, MAPREDUCE-7252.002.patch, > MAPREDUCE-7252.003.patch > > > The simple exponential runtime estimator (added in MAPREDUCE-7208) seems not > to handle the corner cases where the delta progress is 0. As a result, the > forecast will be NaN or Inf which messes up the subsequent forecast values. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7079) TestMRIntermediateDataEncryption is failing in precommit builds
[ https://issues.apache.org/jira/browse/MAPREDUCE-7079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7079: - Attachment: MAPREDUCE-7079.003.patch > TestMRIntermediateDataEncryption is failing in precommit builds > --- > > Key: MAPREDUCE-7079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7079 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jason Darrell Lowe >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7079.001.patch, MAPREDUCE-7079.002.patch, > MAPREDUCE-7079.003.patch > > > TestMRIntermediateDataEncryption is either timing out or tearing down the JVM > which causes the unit tests in jobclient to not pass cleanly during precommit > builds. From sample precommit console output, note the lack of a test results > line when the test is run: > {noformat} > [INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976 > s - in org.apache.hadoop.mapred.TestSequenceFileInputFormat > [INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption > [INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659 > s - in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath > [...] > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 02:14 h > [INFO] Finished at: 2018-04-12T04:27:06+00:00 > [INFO] Final Memory: 24M/594M > [INFO] > > [WARNING] The requested profile "parallel-tests" could not be activated > because it does not exist. > [WARNING] The requested profile "native" could not be activated because it > does not exist. > [WARNING] The requested profile "yarn-ui" could not be activated because it > does not exist. > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on > project hadoop-mapreduce-client-jobclient: There was a timeout or other error > in the fork -> [Help 1] > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7252) Handling 0 progress in SimpleExponential task runtime estimator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles updated MAPREDUCE-7252: -- Fix Version/s: 3.2.2 3.1.4 3.3.0 > Handling 0 progress in SimpleExponential task runtime estimator > --- > > Key: MAPREDUCE-7252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7252 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: MAPREDUCE-7252.001.patch, MAPREDUCE-7252.002.patch, > MAPREDUCE-7252.003.patch > > > The simple exponential runtime estimator (added in MAPREDUCE-7208) seems not > to handle the corner cases where the delta progress is 0. As a result, the > forecast will be NaN or Inf which messes up the subsequent forecast values. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7252) Handling 0 progress in SimpleExponential task runtime estimator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010854#comment-17010854 ] Jonathan Turner Eagles commented on MAPREDUCE-7252: --- [~ahussein], could you provide a specific patch for branch-2.10? Getting this error {noformat} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hadoop-mapreduce-client-app: Compilation failure [ERROR] /Users/jeagles/hadoop/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java:[397,61] method references are not supported in -source 1.7 [ERROR] (use -source 8 or higher to enable method references) {noformat} Which corresponds to this code. {code} String statesValues = targetStates.stream().map(Object::toString).collect( Collectors.joining(",")); {code} > Handling 0 progress in SimpleExponential task runtime estimator > --- > > Key: MAPREDUCE-7252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7252 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: MAPREDUCE-7252.001.patch, MAPREDUCE-7252.002.patch, > MAPREDUCE-7252.003.patch > > > The simple exponential runtime estimator (added in MAPREDUCE-7208) seems not > to handle the corner cases where the delta progress is 0. As a result, the > forecast will be NaN or Inf which messes up the subsequent forecast values. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7252) Handling 0 progress in SimpleExponential task runtime estimator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7252: - Attachment: MAPREDUCE-7252-branch-2.10.003.patch > Handling 0 progress in SimpleExponential task runtime estimator > --- > > Key: MAPREDUCE-7252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7252 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: MAPREDUCE-7252-branch-2.10.003.patch, > MAPREDUCE-7252.001.patch, MAPREDUCE-7252.002.patch, MAPREDUCE-7252.003.patch > > > The simple exponential runtime estimator (added in MAPREDUCE-7208) seems not > to handle the corner cases where the delta progress is 0. As a result, the > forecast will be NaN or Inf which messes up the subsequent forecast values. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7079) TestMRIntermediateDataEncryption is failing in precommit builds
[ https://issues.apache.org/jira/browse/MAPREDUCE-7079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011031#comment-17011031 ] Hadoop QA commented on MAPREDUCE-7079: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 34s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 34s{color} | {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 0 new + 9 unchanged - 24 fixed = 9 total (was 33) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 16s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 32s{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}151m 51s{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 41s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}234m 28s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.mapreduce.TestMRJobClient | | | hadoop.mapreduce.v2.TestUberAM | | | hadoop.mapreduce.security.ssl.TestEncryptedShuffle | | | hadoop.mapreduce.v2.TestMRJobs | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 | | JIRA Issue | MAPREDUCE-7079 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12990231/MAPREDUCE-7079.003.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 32aab82bfcc4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Per
[jira] [Commented] (MAPREDUCE-7252) Handling 0 progress in SimpleExponential task runtime estimator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011050#comment-17011050 ] Hadoop QA commented on MAPREDUCE-7252: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 1s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} branch-2.10 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 1s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 11s{color} | {color:green} branch-2.10 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s{color} | {color:green} branch-2.10 passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s{color} | {color:green} branch-2.10 passed with JDK v1.8.0_232 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 32s{color} | {color:green} branch-2.10 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s{color} | {color:green} branch-2.10 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 23s{color} | {color:green} branch-2.10 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} branch-2.10 passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green} branch-2.10 passed with JDK v1.8.0_232 {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s{color} | {color:green} the patch passed with JDK v1.8.0_232 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 29s{color} | {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 2 new + 64 unchanged - 8 fixed = 66 total (was 72) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} the patch passed with JDK v1.8.0_232 {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 16s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 23s{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 42s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}184m 49s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:a969cad0a12 | | JIRA Issue | MAPREDUCE-7252 | | JIRA Patch URL | http