[Announcement] Apache Hadoop Submarine 0.2.0 released!
Hi folks, I'm glad to announce that the Apache Hadoop community has voted to release Apache Hadoop Submarine 0.2.0. Apache Hadoop Submarine is a project to allows infra engineer/data scientist to run deep learning applications resource management platform (like YARN, K8s). This release contains below new features: 1. Linkedin’s TonY runtime support in Submarine 2. PyTorch enabled in Submarine with both YARN native service runtime (single node) and TonY runtime 3. Support uber jar of Submarine to submit the job 4. The YAML file to describe a job 5. The Notebook support (by Apache Zeppelin Submarine interpreter) For more information and to download, please check: https://hadoop.apache.org/submarine/ For submarine jar files as project dependencies, please check: https://mvnrepository.com/search?q=submarine Tons of thanks to everyone who contributed to the release, and everyone in the Apache Hadoop community! The release is a result of work from many contributors. Thank you for all of them. BR, Zhankun
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1185/ [Jul 1, 2019 5:38:53 AM] (github) HDDS-1730. Implement File CreateDirectory Request to use Cache and Do… [Jul 1, 2019 3:22:32 PM] (msingh) HDDS-1555. Disable install snapshot for ContainerStateMachine. [Jul 1, 2019 3:58:47 PM] (aengineer) HDDS-373. Genconf tool must generate ozone-site.xml with sample values [Jul 1, 2019 5:46:33 PM] (elgoiri) YARN-9661:Fix typo in LocalityMulticastAMRMProxyPolicy.java and [Jul 1, 2019 6:04:32 PM] (elek) HDDS-1258. Fix error propagation for SCM protocol [Jul 1, 2019 7:44:14 PM] (bharat) HDDS-1685. Recon: Add support for "start" query param to containers and [Jul 1, 2019 8:54:49 PM] (aengineer) HDFS-14610. HashMap is not thread safe. Field storageMap is typically [Jul 2, 2019 12:19:57 AM] (elek) HDDS-1667. Docker compose file may referring to incorrect docker image -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore Unread field:TimelineEventSubDoc.java:[line 56] Unread field:TimelineMetricSubDoc.java:[line 44] FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core Class org.apache.hadoop.applications.mawo.server.common.TaskStatus implements Cloneable but does not define or use clone method At TaskStatus.java:does not define or use clone method At TaskStatus.java:[lines 39-346] Equals method for org.apache.hadoop.applications.mawo.server.worker.WorkerId assumes the argument is of type WorkerId At WorkerId.java:the argument is of type WorkerId At WorkerId.java:[line 114] org.apache.hadoop.applications.mawo.server.worker.WorkerId.equals(Object) does not check for null argument At WorkerId.java:null argument At WorkerId.java:[lines 114-115] FindBugs : module:hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra org.apache.hadoop.tools.dynamometer.Client.addFileToZipRecursively(File, File, ZipOutputStream) may fail to clean up java.io.InputStream on checked exception Obligation to clean up resource created at Client.java:to clean up java.io.InputStream on checked exception Obligation to clean up resource created at Client.java:[line 859] is not discharged Exceptional return value of java.io.File.mkdirs() ignored in org.apache.hadoop.tools.dynamometer.DynoInfraUtils.fetchHadoopTarball(File, String, Configuration, Logger) At DynoInfraUtils.java:ignored in org.apache.hadoop.tools.dynamometer.DynoInfraUtils.fetchHadoopTarball(File, String, Configuration, Logger) At DynoInfraUtils.java:[line 138] Found reliance on default encoding in org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(String[]):in org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(String[]): new java.io.InputStreamReader(InputStream) At SimulatedDataNodes.java:[line 149] org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(String[]) invokes System.exit(...), which shuts down the entire virtual machine At SimulatedDataNodes.java:down the entire virtual machine At SimulatedDataNodes.java:[line 123] org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(String[]) may fail to close stream At SimulatedDataNodes.java:stream At SimulatedDataNodes.java:[line 149] FindBugs : module:hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen Self assignment of field BlockInfo.replication in new org.apache.hadoop.tools.dynamometer.blockgenerator.BlockInfo(BlockInfo) At BlockInfo.java:in new org.apache.hadoop.tools.dynamometer.blockgenerator.BlockInfo(BlockInfo) At BlockInfo.java:[line 78] Failed junit tests : hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.client.api.impl.TestAMRMClient hadoop.yarn.sls.TestSLSStreamAMSynth hadoop.ozone.container.common.TestEndPoint hadoop.ozone.scm.TestGetCommittedBlockLengthAndPutKey hadoop.hdds.scm.container.TestContainerStateManagerIntegration hadoop.ozone.web.client.TestKeysRatis hadoop.ozone.scm.pipeline.TestPipelineManagerMXBean hadoop.hdds.scm.safemode.TestSCM
Apache Hadoop Ozone 0.4.1 (Biscayne) release plan
Hello Ozone committers, For Biscayne release, I have cut ozone-0.4.1 branch from trunk and updated "Fix Version/s" for existing resolved jiras accordingly. If you want to include any other jira(s) in this release, please update the "Target Version/s" of the jira(s) and use ozone-0.4.1 branch while committing. I will send out a follow-up mail on the code freeze date. Thanks, Nanda
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/ [Jul 1, 2019 5:59:58 PM] (stack) Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9 [Jul 1, 2019 6:07:43 PM] (stack) Revert "Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to [Jul 1, 2019 6:08:05 PM] (stack) HDFS-14585 Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-common-project/hadoop-common Class org.apache.hadoop.fs.GlobalStorageStatistics defines non-transient non-serializable instance field map In GlobalStorageStatistics.java:instance field map In GlobalStorageStatistics.java FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.datanode.TestBlockScanner hadoop.hdfs.server.datanode.TestFsDatasetCache hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestCachingStrategy hadoop.hdfs.server.datanode.TestBlockRecovery hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/diff-compile-cc-root-jdk1.8.0_212.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/diff-compile-javac-root-jdk1.8.0_212.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/diff-patch-shellcheck.txt [72K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/whitespace-tabs.txt [1.2M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/xml.txt [12K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/370/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-wa
