[jira] [Resolved] (HADOOP-18300) Update Gson to 2.9.0
[ https://issues.apache.org/jira/browse/HADOOP-18300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HADOOP-18300. Fix Version/s: 3.4.0 3.2.4 3.3.9 Hadoop Flags: Reviewed Resolution: Fixed I have committed this to trunk, branch-3.3 and branch-3.2. [~medb], thank you for the contribution. [~ayushtkn], thank you for code reviewing. > Update Gson to 2.9.0 > > > Key: HADOOP-18300 > URL: https://issues.apache.org/jira/browse/HADOOP-18300 > Project: Hadoop Common > Issue Type: Task > Components: build >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.9 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Update to the Gson 2.9.0 that has many > [fixes|https://github.com/google/gson/releases/tag/gson-parent-2.9.0], and > backward-compatible as long as Java 7+ is used. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/ [Jun 21, 2022 8:12:43 AM] (noreply) HDFS-16637. TestHDFSCLI#testAll consistently failing (#4466). Contributed by Viraj Jasani. [Jun 21, 2022 1:54:11 PM] (noreply) MAPREDUCE-7389. Fix typo in description of property (#4440). Contributed by Christian Bartolomaus. -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.mapred.TestLocalDistributedCacheManager hadoop.yarn.server.router.webapp.TestRouterWebServicesREST cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/results-compile-javac-root.txt [540K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/results-javadoc-javadoc-root.txt [400K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt [48K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/899/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt [84K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Merged HADOOP-18103 (Vectored IO) with a merge commit in trunk.
Hi Everyone, Today we merged Vectored IO feature ( https://issues.apache.org/jira/browse/HADOOP-18103) from https://github.com/apache/hadoop/tree/feature-vectored-io feature branch into apache trunk and included a merge commit to the top containing all the details of child commits. We will be cherry-picking the individual commits and backport to branch-3.3. Me and @Steve Loughran are planning to do a big 3.4.0 release of Hadoop in Q32022 which is going to include the vectored io feature.
Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.2-java8-linux-x86_64/52/ [Jun 20, 2022 2:40:11 AM] (Akira Ajisaka) HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) [Error replacing 'FILE' - Workspace is not accessible] - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18237) Upgrade Apache Xerces Java to 2.12.2
[ https://issues.apache.org/jira/browse/HADOOP-18237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18237. - Fix Version/s: 3.4.0 3.3.4 Resolution: Fixed > Upgrade Apache Xerces Java to 2.12.2 > > > Key: HADOOP-18237 > URL: https://issues.apache.org/jira/browse/HADOOP-18237 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.4 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Description > https://github.com/advisories/GHSA-h65f-jvqw-m9fj > There's a vulnerability within the Apache Xerces Java (XercesJ) XML parser > when handling specially crafted XML document payloads. This causes, the > XercesJ XML parser to wait in an infinite loop, which may sometimes consume > system resources for prolonged duration. This vulnerability is present within > XercesJ version 2.12.1 and the previous versions. > References > [https://nvd.nist.gov/vuln/detail/CVE-2022-23437] > https://lists.apache.org/thread/6pjwm10bb69kq955fzr1n0nflnjd27dl > http://www.openwall.com/lists/oss-security/2022/01/24/3 > https://www.oracle.com/security-alerts/cpuapr2022.html -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18311) Upgrade dependencies to address several CVEs
Steve Vaughan created HADOOP-18311: -- Summary: Upgrade dependencies to address several CVEs Key: HADOOP-18311 URL: https://issues.apache.org/jira/browse/HADOOP-18311 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.3.3, 3.3.4 Reporter: Steve Vaughan Fix For: 3.3.4 The following CVEs can be addressed by upgrading dependencies within the build. This includes a replacement of HTrace with a noop implementation. * CVE-2018-7489 * CVE-2020-10663 * CVE-2020-28491 * CVE-2020-35490 * CVE-2020-35491 * CVE-2020-36518 * PRISMA-2021-0182 This addresses all of the CVEs from 3.3.3 except for ones that would require upgrading Netty to 4.x. I'll be submitting a pull request for 3.3.4. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18293) Release Hadoop 3.3.4 critical fix update
[ https://issues.apache.org/jira/browse/HADOOP-18293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18293. - Resolution: Duplicate forgot about this when i created HADOOP-18305; closing > Release Hadoop 3.3.4 critical fix update > > > Key: HADOOP-18293 > URL: https://issues.apache.org/jira/browse/HADOOP-18293 > Project: Hadoop Common > Issue Type: Task > Components: build >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Create a new release off the branch-3.3.3 line with a few more changes > * wrap up of security changes > * cut hadoop-cos out of hadoop-cloud-storage as its dependencies break s3a > client...reinstate once the updated jar is tested > * try to get an arm build out tool -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Hadoop 3.3.4 release process underway
update 1. branch 3.3 has a version of 3.3.9-SNAPSHOT 2. and 3.3.9 should be the version for fixes against this, with 3.3.4 for the new point release 3. please don't use 3.3.4 for branch-3.3 changes from now on I've already got a PR up with the changes; going to create an asf branch-3.3.4 branch mirroring it and kicking off with the pom update. I'm going to do a dry run of a release this week to build the binaries on x86 and ARM but not put for a vote as I am away next week. instead it'll be a validation of my processes, ant-based automation etc. ( https://github.com/steveloughran/validate-hadoop-client-artifacts) I will kick off the release the following week, which, being july4 week, may have more people offline. it does give larry a chance to get https://issues.apache.org/jira/browse/HADOOP-18074 in, as it may have security implications. On Mon, 20 Jun 2022 at 19:02, Steve Loughran wrote: > I'm setting things up for a new release > > https://issues.apache.org/jira/browse/HADOOP-18305 > > absolute minimum of fixes. as well as some related to ZK lockdown, i would > like to include > > https://issues.apache.org/jira/browse/HADOOP-18303 > remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime > > and > https://issues.apache.org/jira/browse/HADOOP-18307 > remove hadoop-cos as a dependency of hadoop-cloud-storage > > the last one is a lstminute workaround for, a classpath ordering issue due > to too many libraries having unshaded references to mozilla/prefix-list.txt > https://issues.apache.org/jira/browse/HADOOP-18159 > > the proper fixes would be getting that updated library tested (who is set > up to test tencent cloud?) and ideally (aws, cos, gcs shaded libraries to > shade their references) > > for 3.3.4, i am just going to cut the declaration of the module as a > dependency of hadoop-cloud-storage so out of downstream apps unless they > explicitly ask forit. > > now, numbering > > >1. I am calling this 3.3.4 >2. I am going to increase the version of branch 3.3. to 3.3.9. that >leaves space for some more but doesn't confuse jira dropdown dialogs. > > > i do believe branch-3.3. should be renamed branch-3.4 and the release i > plan to do with mukund called 3.4.0, but that is another bit of project > organisation. > > expect the first RC up soon, I am going to be away on vacation from june > 28 to july 23 though, which complicates things > > >
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapred.TestLineRecordReader hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-mvnsite-root.txt [568K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [428K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [132K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K] asflicense: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/700/artifact/out/patch-asflicense-p
