Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes hadoop.hdfs.server.balancer.TestBalancer hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/patch-mvnsite-root.txt [572K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/patch-javadoc-root.txt [36K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [532K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1267/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] ht
[jira] [Created] (HADOOP-19032) MultiObjectDeleteException bulk delete of odd filenames
Steve Loughran created HADOOP-19032:
---
Summary: MultiObjectDeleteException bulk delete of odd filenames
Key: HADOOP-19032
URL: https://issues.apache.org/jira/browse/HADOOP-19032
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Possibly transient. note bucket is versioned.
{code}
org.apache.hadoop.fs.s3a.AWSS3IOException:
Remove S3 Dir Markers on
s3a://stevel-london/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest:
org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteException:
[S3Error(Key=Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest/()&^%$#@!~_+}{>
[jira] [Created] (HADOOP-19033) S3A: disable checksum validation
Steve Loughran created HADOOP-19033: --- Summary: S3A: disable checksum validation Key: HADOOP-19033 URL: https://issues.apache.org/jira/browse/HADOOP-19033 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Steve Loughran Assignee: Steve Loughran AWS v2 sdk turns on client-side checksum validation; this kills performance Given we are using TLS to download from AWS s3, there's implicit channel checksumming going on on, that's along with the IPv4 TCP checksumming. We don't need it, all it does is slow us down. proposed: disable in DefaultS3ClientFactory I don't want to add an option to enable it as it only complicates life (yet another config option), but I am open to persuasion -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/ No changes -1 overall The following subsystems voted -1: blanks hadolint pathlen spotbugs xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-yarn-project/hadoop-yarn org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT isn't final but should be At TimelineConnector.java:be At TimelineConnector.java:[line 82] spotbugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT isn't final but should be At TimelineConnector.java:be At TimelineConnector.java:[line 82] spotbugs : module:hadoop-yarn-project org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT isn't final but should be At TimelineConnector.java:be At TimelineConnector.java:[line 82] spotbugs : module:root org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT isn't final but should be At TimelineConnector.java:be At TimelineConnector.java:[line 82] cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/results-compile-javac-root.txt [12K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/blanks-eol.txt [15M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/results-checkstyle-root.txt [13M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/results-hadolint.txt [24K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/results-shellcheck.txt [24K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/results-javadoc-javadoc-root.txt [244K] spotbugs: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn-warnings.html [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html [8.0K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/branch-spotbugs-hadoop-yarn-project-warnings.html [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1467/artifact/out/branch-spotbugs-root-warnings.html [20K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[VOTE] Release Apache Hadoop 3.4.0 RC0
Hello all, We plan to release hadoop 3.4.0 based on hadoop trunk, which is the first hadoop 3.4.0-RC version. The RC is available at: https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ (for amd64) https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-arm64/ (for arm64) Maven artifacts is built by x86 machine and are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1391/ My public key: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Changelog: https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/CHANGELOG.md Release notes: https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/RELEASENOTES.md This is a relatively big release (by Hadoop standard) containing about 2852 commits. Please give it a try, this RC vote will run for 7 days. Feature highlights: DataNode FsDatasetImpl Fine-Grained Locking via BlockPool [HDFS-15180](https://issues.apache.org/jira/browse/HDFS-15180) Split FsDatasetImpl datasetLock via blockpool to solve the issue of heavy FsDatasetImpl datasetLock When there are many namespaces in a large cluster. YARN Federation improvements [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) brings many improvements, including the following: 1. YARN Router now boasts a full implementation of all relevant interfaces including the ApplicationClientProtocol, ResourceManagerAdministrationProtocol, and RMWebServiceProtocol. 2. Enhanced support for Application cleanup and automatic offline mechanisms for SubCluster are now facilitated by the YARN Router. 3. Code optimization for Router and AMRMProxy was undertaken, coupled with improvements to previously pending functionalities. 4. Audit logs and Metrics for Router received upgrades. 5. A boost in cluster security features was achieved, with the inclusion of Kerberos support. 6. The page function of the router has been enhanced. Upgrade AWS SDK to V2 [HADOOP-18073](https://issues.apache.org/jira/browse/HADOOP-18073) The S3A connector now uses the V2 AWS SDK. This is a significant change at the source code level. Any applications using the internal extension/override points in the filesystem connector are likely to break. Consult the document aws\_sdk\_upgrade for the full details. hadoop-thirdparty will also provide the new RC0 soon. Best Regards, Shilun Fan.
Re: [VOTE] Release Apache Hadoop 3.4.0 RC0
Thanks for driving this release, Shilun Fan. The top page of site documentation (in hadoop-3.4.0-RC0-site.tar.gz) looks the same as 3.3.5. While the index.md.vm is updated in branch-3.4.0[1], it seems not to be reflected. release-3.4.0-RC0 tag should be pushed to make checking easier. In addition, the description about new features of previous release should be removed from the index.md.vm. [1] https://github.com/apache/hadoop/blob/branch-3.4.0/hadoop-project/src/site/markdown/index.md.vm Masatake Iwasaki On 2024/01/11 14:15, slfan1989 wrote: Hello all, We plan to release hadoop 3.4.0 based on hadoop trunk, which is the first hadoop 3.4.0-RC version. The RC is available at: https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ (for amd64) https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-arm64/ (for arm64) Maven artifacts is built by x86 machine and are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1391/ My public key: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Changelog: https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/CHANGELOG.md Release notes: https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/RELEASENOTES.md This is a relatively big release (by Hadoop standard) containing about 2852 commits. Please give it a try, this RC vote will run for 7 days. Feature highlights: DataNode FsDatasetImpl Fine-Grained Locking via BlockPool [HDFS-15180](https://issues.apache.org/jira/browse/HDFS-15180) Split FsDatasetImpl datasetLock via blockpool to solve the issue of heavy FsDatasetImpl datasetLock When there are many namespaces in a large cluster. YARN Federation improvements [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) brings many improvements, including the following: 1. YARN Router now boasts a full implementation of all relevant interfaces including the ApplicationClientProtocol, ResourceManagerAdministrationProtocol, and RMWebServiceProtocol. 2. Enhanced support for Application cleanup and automatic offline mechanisms for SubCluster are now facilitated by the YARN Router. 3. Code optimization for Router and AMRMProxy was undertaken, coupled with improvements to previously pending functionalities. 4. Audit logs and Metrics for Router received upgrades. 5. A boost in cluster security features was achieved, with the inclusion of Kerberos support. 6. The page function of the router has been enhanced. Upgrade AWS SDK to V2 [HADOOP-18073](https://issues.apache.org/jira/browse/HADOOP-18073) The S3A connector now uses the V2 AWS SDK. This is a significant change at the source code level. Any applications using the internal extension/override points in the filesystem connector are likely to break. Consult the document aws\_sdk\_upgrade for the full details. hadoop-thirdparty will also provide the new RC0 soon. Best Regards, Shilun Fan. - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
