[jira] [Created] (MAPREDUCE-6971) Moving logging APIs over to slf4j in hadoop-mapreduce-client-app
Jinjiang Ling created MAPREDUCE-6971: Summary: Moving logging APIs over to slf4j in hadoop-mapreduce-client-app Key: MAPREDUCE-6971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6971 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Jinjiang Ling -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/ [Sep 26, 2017 7:46:03 AM] (aajisaka) HADOOP-14893. WritableRpcEngine should use Time.monotonicNow. [Sep 26, 2017 9:04:09 AM] (rohithsharmaks) YARN-65. Reduce RM app memory footprint once app has completed. [Sep 26, 2017 9:21:42 AM] (yqlin) HDFS-12495. TestPendingInvalidateBlock#testPendingDeleteUnknownBlocks [Sep 26, 2017 3:38:03 PM] (jlowe) Revert "YARN-6570. No logs were found for running application, running" -1 overall The following subsystems voted -1: unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.crypto.key.TestValueQueue hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem hadoop.crypto.key.kms.server.TestKMS hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.TestSafeModeWithStripedFile hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer hadoop.yarn.server.router.webapp.TestRouterWebServicesREST hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapreduce.TestMapreduceConfigFields hadoop.mapreduce.v2.hs.webapp.TestHSWebApp hadoop.yarn.sls.TestReservationSystemInvariants hadoop.yarn.sls.TestSLSRunner Timed out junit tests : org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-compile-javac-root.txt [292K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/whitespace-tabs.txt [1.2M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/diff-javadoc-javadoc-root.txt [1.9M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [156K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-common-project_hadoop-kms.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [352K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [64K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt [604K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/540/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [24K] Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6970) archive-logs tool should throttle container requests
Robert Kanter created MAPREDUCE-6970:
Summary: archive-logs tool should throttle container requests
Key: MAPREDUCE-6970
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6970
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0-alpha1, 2.8.0
Reporter: Robert Kanter
The {{mapred archive-logs}} command currently has no way to throttle the number
of requested containers. For example, we recently saw a busy cluster where the
tool hadn't been run for a while and there were about 20,000 apps to process.
This meant that the tool tried to request 20,000 containers and got a ton of GC
and then OOM trying to handle that.
This problem can be mitigated by setting {{-maxEligibleApps}} to a more
reasonable value, but doing so would require running the tool multiple times;
plus, the default value is {{-1}} (all).
We should add a way to throttle the max number of concurrently running
containers that the tool manages. Something like {{-concurrency }} where it
would only allow up to {{n}} containers at a time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6969) TestHSWebApp is failing
Jason Lowe created MAPREDUCE-6969:
-
Summary: TestHSWebApp is failing
Key: MAPREDUCE-6969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Jason Lowe
TestHSWebApp has been failing recently:
{noformat}
Running org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp
Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.57 sec <<<
FAILURE! - in org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp
testLogsViewBadStartEnd(org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp)
Time elapsed: 0.076 sec <<< FAILURE!
org.mockito.exceptions.verification.junit.ArgumentsAreDifferent:
Argument(s) are different! Wanted:
printWriter.write(
"Invalid log end value: bar"
);
-> at
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp.testLogsViewBadStartEnd(TestHSWebApp.java:261)
Actual invocation has different arguments:
printWriter.write(
"http://www.w3.org/TR/html4/strict.dtd";>"
);
-> at
org.apache.hadoop.yarn.webapp.view.TextView.echoWithoutEscapeHtml(TextView.java:62)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp.testLogsViewBadStartEnd(TestHSWebApp.java:261)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6968) Staging directory erasure coding config property has a typo
Jason Lowe created MAPREDUCE-6968: - Summary: Staging directory erasure coding config property has a typo Key: MAPREDUCE-6968 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6968 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.0.0-beta1 Reporter: Jason Lowe Assignee: Jason Lowe TestMapreduceConfigFields has been failing since MAPREDUCE-6954. MRJobConfig#MR_AM_STAGING_DIR_ERASURECODING_ENABLED is defined as "yarn.app.mapreduce.am.staging-direrasurecoding.enabled" but the property is listed as "yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled" in mapred-default.xml. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/ [Sep 25, 2017 12:12:25 PM] (templedf) HADOOP-14897. Loosen compatibility guidelines for native dependencies [Sep 25, 2017 4:25:07 PM] (aajisaka) HDFS-12304. Remove unused parameter from FsDatasetImpl#addVolume. [Sep 25, 2017 4:47:02 PM] (aajisaka) HADOOP-14892. MetricsSystemImpl should use Time.monotonicNow for [Sep 25, 2017 5:03:30 PM] (aajisaka) YARN-7153. Remove duplicated code in AMRMClientAsyncImpl.java. [Sep 25, 2017 8:35:44 PM] (jlowe) HADOOP-14881. LoadGenerator should use Time.monotonicNow() to measure [Sep 25, 2017 8:55:09 PM] (aengineer) HDFS-12529. Get source for config tags from file name. Contributed by [Sep 25, 2017 10:30:51 PM] (rchiang) HADOOP-14903. Add json-smart explicitly to pom.xml. (rchiang) [Sep 25, 2017 10:59:38 PM] (fabbri) HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity [Sep 25, 2017 11:02:55 PM] (Arun Suresh) YARN-7240. Add more states and transitions to stabilize the NM Container [Sep 26, 2017 12:12:01 AM] (fabbri) HADOOP-14890 Move up to AWS SDK 1.11.199. Contributed by Steve Loughran. -1 overall The following subsystems voted -1: unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation hadoop.yarn.client.api.impl.TestAMRMClient hadoop.mapreduce.TestMapreduceConfigFields hadoop.mapreduce.v2.hs.webapp.TestHSWebApp hadoop.yarn.sls.TestSLSRunner hadoop.yarn.sls.TestReservationSystemInvariants Timed out junit tests : org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-compile-javac-root.txt [292K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/whitespace-tabs.txt [1.2M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/diff-javadoc-javadoc-root.txt [1.9M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [64K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/539/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [20K] Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
