Re: [DISCUSS] Release Hadoop 3.4.0
Dear Community Members, While discussing release Hadoop 3.4.0, we have also decided to release Hadoop Thirdparty version 1.2.0. I'll be initiating some branch preparation work over the weekend. Best Regards, Shilun Fan. -- Forwarded message - 发件人： slfan1989 Date: 2024/01/13 08:08:36 Subject: Re: Re: [DISCUSS] Release Hadoop 3.4.0 To: , , < ste...@cloudera.com> Cc: , Ayush Saxena , < hexiaoq...@apache.org> Thank you for reporting this issue and helping resolve it！ I will try to release hadoop-thirdparty 1.2.0-RC0 version first. Best Regards, Shilun Fan. Original From:"PJ Fanning"< fannin...@apache.org >; Date:2024/1/9 1:49 To:"common-dev"< common-dev@hadoop.apache.org >; Subject:Re: Re: [DISCUSS] Release Hadoop 3.4.0 It looks like the Jackson upgrade [1] needs more investigation which I will need to come back to in a few weeks. I would still like to get hadoop-thirdparty 1.2.0 released so that protobuf-java, avro and guava can be upgraded for hadoop 3.4.0. [1] https://github.com/apache/hadoop/pull/6370 On 2024/01/04 15:53:43 PJ Fanning wrote: > I would like to get some dependencies upgraded for Hadoop 3.4.0. For me, it would be good to upgrade protobuf-java and Jackson to more secure versions. > > For protobuf-java, that would involve releasing hadoop-thirdparty 1.2.0 [1], including merging the protobuf upgrade [2]. > > For Jackson, we are hampered by Jackson dropping support for Jersey/JAX-RS 1.x. I have a workaround for this that I think is worth further investigating [3]. > > [1] https://github.com/apache/hadoop-thirdparty > [2] https://github.com/apache/hadoop-thirdparty/pull/19 > [3] https://github.com/apache/hadoop/pull/6370 > > > > On 2024/01/04 14:26:37 slfan1989 wrote: > > Hey all, > > > > We are planning to release Hadoop 3.4.0 base on trunk. I made some > > preparations and changed the target version of JIRA for non-blockers in > > HADOOP, HDFS, YARN, and MAPREDUCE from 3.4.0 to 3.5.0. If we want to create > > a new JIRA, the target version can directly select version 3.5.0. > > > > If you have any thoughts, suggestions, or concerns, please feel free to > > share them. > > > > Best Regards, > > Shilun Fan. > > > > > +1 from me. > > >> It will include the new AWS V2 SDK upgrade as well. > > > > > On Wed, Jan 3, 2024 at 6:35 AM Xiaoqiao He wrote: > > > > > > > > > > I think the release discussion can be in public ML? > > > > > > Good idea. cc common-dev/hdfs-dev/yarn-dev/mapreduce-dev ML. > > > > > > Best Regards, > > > - He Xiaoqiao > > > > > > On Tue, Jan 2, 2024 at 6:18 AM Ayush Saxena wrote: > > > > > > > +1 from me as well. > > > > > > > > We should definitely attempt to upgrade the thirdparty version for > > > > 3.4.0 & check if there are any pending critical/blocker issues as > > > > well. > > > > > > > > I think the release discussion can be in public ML? > > > > > > > > -Ayush > > > > > > > > On Mon, 1 Jan 2024 at 18:25, Steve Loughran > > > > > > > wrote: > > > > > > > > > > +1 from me > > > > > > > > > > ant and maven repo to build and validate things, including making arm > > > > > binaries if you work from an arm macbook. > > > > > https://github.com/steveloughran/validate-hadoop-client-artifacts > > > > > > > > > > do we need to publish an up to date thirdparty release for this? > > > > > > > > > > > > > > > > > > > > On Mon, 25 Dec 2023 at 16:06, slfan1989 wrote: > > > > > > > > > > > Dear PMC Members, > > > > > > > > > > > > First of all, Merry Christmas to everyone! > > > > > > > > > > > > In our community discussions, we collectively finalized the plan to > > > > release > > > > > > Hadoop 3.4.0 based on the current trunk branch. I am applying to > > take > > > > on > > > > > > the responsibility for the initial release of version 3.4.0, and the > > > > entire > > > > > > process is set to officially commence in January 2024. > > > > > > I have created a new JIRA: HADOOP-19018. Release 3.4.0. > > > > > > > > > > > > The specific work plan includes: > > > > > > > > > > > > 1. Following the guidance in the HowToRelease document, completing > > > all > > > > the > > > > > > relevant tasks required for the release of version 3.4.0. > > > > > > 2. Pointing the trunk branch to 3.5.0-SNAPSHOT. > > > > > > 3. Currently, the Fix Versions of all tasks merged into trunk are > > set > > > > as > > > > > > 3.4.0; I will move them to 3.5.0. > > > > > > > > > > > > Confirmed features to be included in the release: > > > > > > > > > > > > 1. Enhanced functionality for YARN Federation. > > > > > > 2. Optimization of HDFS RBF. > > > > > > 3. Introduction of fine-grained global locks for DataNodes. > > > > > > 4. Improvements in the stability of HDFS EC, and more. > > > > > > 5. Fixes for important CVEs. > > > > > > > > > > > > If you have any thoughts, suggestions, or concerns, please feel free > > > to > > > > > > share them. > > > > > > > > > > > > Looking forward to a successful release! > > > > > > > > > > > > Best Regards, > > > > > > Shilun Fan. > > > > >
Re: Re: [DISCUSS] Release Hadoop 3.4.0
Thank you for reporting this issue and helping resolve it！ I will try to release hadoop-thirdparty 1.2.0-RC0 version first. Best Regards, Shilun Fan. Original From:"PJ Fanning"< fannin...@apache.org >; Date:2024/1/9 1:49 To:"common-dev"< common-dev@hadoop.apache.org >; Subject:Re: Re: [DISCUSS] Release Hadoop 3.4.0 It looks like the Jackson upgrade [1] needs more investigation which I will need to come back to in a few weeks. I would still like to get hadoop-thirdparty 1.2.0 released so that protobuf-java, avro and guava can be upgraded for hadoop 3.4.0. [1] https://github.com/apache/hadoop/pull/6370 On 2024/01/04 15:53:43 PJ Fanning wrote: > I would like to get some dependencies upgraded for Hadoop 3.4.0. For me, it would be good to upgrade protobuf-java and Jackson to more secure versions. > > For protobuf-java, that would involve releasing hadoop-thirdparty 1.2.0 [1], including merging the protobuf upgrade [2]. > > For Jackson, we are hampered by Jackson dropping support for Jersey/JAX-RS 1.x. I have a workaround for this that I think is worth further investigating [3]. > > [1] https://github.com/apache/hadoop-thirdparty > [2] https://github.com/apache/hadoop-thirdparty/pull/19 > [3] https://github.com/apache/hadoop/pull/6370 > > > > On 2024/01/04 14:26:37 slfan1989 wrote: > > Hey all, > > > > We are planning to release Hadoop 3.4.0 base on trunk. I made some > > preparations and changed the target version of JIRA for non-blockers in > > HADOOP, HDFS, YARN, and MAPREDUCE from 3.4.0 to 3.5.0. If we want to create > > a new JIRA, the target version can directly select version 3.5.0. > > > > If you have any thoughts, suggestions, or concerns, please feel free to > > share them. > > > > Best Regards, > > Shilun Fan. > > > > > +1 from me. > > >> It will include the new AWS V2 SDK upgrade as well. > > > > > On Wed, Jan 3, 2024 at 6:35 AM Xiaoqiao He wrote: > > > > > > > > > > I think the release discussion can be in public ML? > > > > > > Good idea. cc common-dev/hdfs-dev/yarn-dev/mapreduce-dev ML. > > > > > > Best Regards, > > > - He Xiaoqiao > > > > > > On Tue, Jan 2, 2024 at 6:18 AM Ayush Saxena wrote: > > > > > > > +1 from me as well. > > > > > > > > We should definitely attempt to upgrade the thirdparty version for > > > > 3.4.0 & check if there are any pending critical/blocker issues as > > > > well. > > > > > > > > I think the release discussion can be in public ML? > > > > > > > > -Ayush > > > > > > > > On Mon, 1 Jan 2024 at 18:25, Steve Loughran > > > > > > > wrote: > > > > > > > > > > +1 from me > > > > > > > > > > ant and maven repo to build and validate things, including making arm > > > > > binaries if you work from an arm macbook. > > > > > https://github.com/steveloughran/validate-hadoop-client-artifacts > > > > > > > > > > do we need to publish an up to date thirdparty release for this? > > > > > > > > > > > > > > > > > > > > On Mon, 25 Dec 2023 at 16:06, slfan1989 wrote: > > > > > > > > > > > Dear PMC Members, > > > > > > > > > > > > First of all, Merry Christmas to everyone! > > > > > > > > > > > > In our community discussions, we collectively finalized the plan to > > > > release > > > > > > Hadoop 3.4.0 based on the current trunk branch. I am applying to > > take > > > > on > > > > > > the responsibility for the initial release of version 3.4.0, and the > > > > entire > > > > > > process is set to officially commence in January 2024. > > > > > > I have created a new JIRA: HADOOP-19018. Release 3.4.0. > > > > > > > > > > > > The specific work plan includes: > > > > > > > > > > > > 1. Following the guidance in the HowToRelease document, completing > > > all > > > > the > > > > > > relevant tasks required for the release of version 3.4.0. > > > > > > 2. Pointing the trunk branch to 3.5.0-SNAPSHOT. > > > > > > 3. Currently, the Fix Versions of all tasks merged into trunk are > > set > > > > as > > > > > > 3.4.0; I will move them to 3.5.0. > > > > > > > > > > > > Confirmed features to be included in the release: > > > > > > > > > > > > 1. Enhanced functionality for YARN Federation. > > > > > > 2. Optimization of HDFS RBF. > > > > > > 3. Introduction of fine-grained global locks for DataNodes. > > > > > > 4. Improvements in the stability of HDFS EC, and more. > > > > > > 5. Fixes for important CVEs. > > > > > > > > > > > > If you have any thoughts, suggestions, or concerns, please feel free > > > to > > > > > > share them. > > > > > > > > > > > > Looking forward to a successful release! > > > > > > > > > > > > Best Regards, > > > > > > Shilun Fan. > > > > > > > > > > > > > > > > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail:
Re: [VOTE] Release Apache Hadoop 3.4.0 RC0
Thank you for reporting this issue and I will follow up on HDFS-17129. I acknowledge that I haven't checked for those marked as critical/blocker in RC0. I intend to complete this check before the release of RC1. Thank you again for your valuable suggestions! Best Regards Shilun Fan. Ayush Saxena 于2024年1月13日周六 05:10写道： > We should consider including > https://issues.apache.org/jira/browse/HDFS-17129 > > Which looks like inducing some misorder between IBR & FBR which can > potentially lead to strange issues, if that can’t be merged, should revert > the one which causes that. > > I think we should check for any ticket which has a target version or > affect version & is marked critica/blockerl for 3.4.0 before spinning up a > new RC, I think I mentioned that somewhere before. > > -1, in case HDFS-17129 is not a false alarm or we can prove it won't cause > any issues. There is a comment which says a block was reported missing post > the patch that induced it: [1] > > [1] https://github.com/apache/hadoop/pull/6244#issuecomment-1793981740 > > -Ayush > > > On Fri, 12 Jan 2024 at 07:37, slfan1989 wrote: > >> Thank you very much for your help in verifying this version! We will use >> version 3.5.0 for fix jira in the future. >> >> Best Regards, >> Shilun Fan. >> >> > wonderful! I'll be testing over the weekend >> >> > Meanwhile, new changes I'm putting in to trunk are tagged as fixed in >> 3.5.0 >> > -correct? >> >> > steve >> >> >> > On Thu, 11 Jan 2024 at 05:15, slfan1989 wrote: >> >> > Hello all, >> > >> > We plan to release hadoop 3.4.0 based on hadoop trunk, which is the >> first >> > hadoop 3.4.0-RC version. >> > >> > The RC is available at: >> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ (for amd64) >> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-arm64/ (for arm64) >> > >> > Maven artifacts is built by x86 machine and are staged at >> > >> https://repository.apache.org/content/repositories/orgapachehadoop-1391/ >> > >> > My public key: >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS >> > >> > Changelog: >> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/CHANGELOG.md >> > >> > Release notes: >> > >> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/RELEASENOTES.md >> > >> > This is a relatively big release (by Hadoop standard) containing about >> 2852 >> > commits. >> > >> > Please give it a try, this RC vote will run for 7 days. >> > >> > Feature highlights: >> > >> > DataNode FsDatasetImpl Fine-Grained Locking via BlockPool >> > >> > [HDFS-15180](https://issues.apache.org/jira/browse/HDFS-15180) Split >> > FsDatasetImpl datasetLock via blockpool to solve the issue of heavy >> > FsDatasetImpl datasetLock >> > When there are many namespaces in a large cluster. >> > >> > YARN Federation improvements >> > >> > [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) brings >> many >> > improvements, including the following: >> > >> > 1. YARN Router now boasts a full implementation of all relevant >> interfaces >> > including the ApplicationClientProtocol, >> > ResourceManagerAdministrationProtocol, and RMWebServiceProtocol. >> > 2. Enhanced support for Application cleanup and automatic offline >> > mechanisms for SubCluster are now facilitated by the YARN Router. >> > 3. Code optimization for Router and AMRMProxy was undertaken, coupled >> with >> > improvements to previously pending functionalities. >> > 4. Audit logs and Metrics for Router received upgrades. >> > 5. A boost in cluster security features was achieved, with the inclusion >> of >> > Kerberos support. >> > 6. The page function of the router has been enhanced. >> > >> > Upgrade AWS SDK to V2 >> > >> > [HADOOP-18073](https://issues.apache.org/jira/browse/HADOOP-18073) >> > The S3A connector now uses the V2 AWS SDK. This is a significant change >> at >> > the source code level. >> > Any applications using the internal extension/override points in the >> > filesystem connector are likely to break. >> > Consult the document aws\_sdk\_upgrade for the full details. >> > >> > hadoop-thirdparty will also provide the new RC0 soon. >> > >> > Best Regards, >> > Shilun Fan. >> > >> >
Re: [VOTE] Release Apache Hadoop 3.4.0 RC0
We should consider including https://issues.apache.org/jira/browse/HDFS-17129 Which looks like inducing some misorder between IBR & FBR which can potentially lead to strange issues, if that can’t be merged, should revert the one which causes that. I think we should check for any ticket which has a target version or affect version & is marked critica/blockerl for 3.4.0 before spinning up a new RC, I think I mentioned that somewhere before. -1, in case HDFS-17129 is not a false alarm or we can prove it won't cause any issues. There is a comment which says a block was reported missing post the patch that induced it: [1] [1] https://github.com/apache/hadoop/pull/6244#issuecomment-1793981740 -Ayush On Fri, 12 Jan 2024 at 07:37, slfan1989 wrote: > Thank you very much for your help in verifying this version! We will use > version 3.5.0 for fix jira in the future. > > Best Regards, > Shilun Fan. > > > wonderful! I'll be testing over the weekend > > > Meanwhile, new changes I'm putting in to trunk are tagged as fixed in > 3.5.0 > > -correct? > > > steve > > > > On Thu, 11 Jan 2024 at 05:15, slfan1989 wrote: > > > Hello all, > > > > We plan to release hadoop 3.4.0 based on hadoop trunk, which is the first > > hadoop 3.4.0-RC version. > > > > The RC is available at: > > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ (for amd64) > > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-arm64/ (for arm64) > > > > Maven artifacts is built by x86 machine and are staged at > > https://repository.apache.org/content/repositories/orgapachehadoop-1391/ > > > > My public key: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > Changelog: > > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/CHANGELOG.md > > > > Release notes: > > > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/RELEASENOTES.md > > > > This is a relatively big release (by Hadoop standard) containing about > 2852 > > commits. > > > > Please give it a try, this RC vote will run for 7 days. > > > > Feature highlights: > > > > DataNode FsDatasetImpl Fine-Grained Locking via BlockPool > > > > [HDFS-15180](https://issues.apache.org/jira/browse/HDFS-15180) Split > > FsDatasetImpl datasetLock via blockpool to solve the issue of heavy > > FsDatasetImpl datasetLock > > When there are many namespaces in a large cluster. > > > > YARN Federation improvements > > > > [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) brings many > > improvements, including the following: > > > > 1. YARN Router now boasts a full implementation of all relevant > interfaces > > including the ApplicationClientProtocol, > > ResourceManagerAdministrationProtocol, and RMWebServiceProtocol. > > 2. Enhanced support for Application cleanup and automatic offline > > mechanisms for SubCluster are now facilitated by the YARN Router. > > 3. Code optimization for Router and AMRMProxy was undertaken, coupled > with > > improvements to previously pending functionalities. > > 4. Audit logs and Metrics for Router received upgrades. > > 5. A boost in cluster security features was achieved, with the inclusion > of > > Kerberos support. > > 6. The page function of the router has been enhanced. > > > > Upgrade AWS SDK to V2 > > > > [HADOOP-18073](https://issues.apache.org/jira/browse/HADOOP-18073) > > The S3A connector now uses the V2 AWS SDK. This is a significant change > at > > the source code level. > > Any applications using the internal extension/override points in the > > filesystem connector are likely to break. > > Consult the document aws\_sdk\_upgrade for the full details. > > > > hadoop-thirdparty will also provide the new RC0 soon. > > > > Best Regards, > > Shilun Fan. > > >
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/ [Jan 11, 2024, 12:01:59 PM] (github) YARN-10125. [Federation] Kill application from client does not kill Unmanaged AM's and containers launched by Unmanaged AM. (#6363) Contributed by Shilun Fan. [Jan 11, 2024, 12:47:00 PM] (github) YARN-11646. Do not ignore zero memory capacity config in QueueCapacityConfigParser. (#6433) [Jan 11, 2024, 1:08:37 PM] (github) HDFS-17312. packetsReceived metric should ignore heartbeat packet. (#6394) [Jan 11, 2024, 1:51:59 PM] (github) MAPREDUCE-7468: Change add-opens flag's default value from true to false (#6436) [Jan 11, 2024, 2:06:15 PM] (github) HADOOP-18981. Move oncrpc and portmap packages to hadoop-common (#6280) [Jan 11, 2024, 5:13:31 PM] (github) HADOOP-19004. S3A: Support Authentication through HttpSigner API (#6324) -1 overall The following subsystems voted -1: blanks hadolint pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-yarn-project/hadoop-yarn org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT isn't final but should be At TimelineConnector.java:be At TimelineConnector.java:[line 82] spotbugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT isn't final but should be At TimelineConnector.java:be At TimelineConnector.java:[line 82] spotbugs : module:hadoop-yarn-project org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT isn't final but should be At TimelineConnector.java:be At TimelineConnector.java:[line 82] spotbugs : module:root org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT isn't final but should be At TimelineConnector.java:be At TimelineConnector.java:[line 82] Failed junit tests : hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.mapreduce.v2.app.job.impl.TestMapReduceChildJVM cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/results-compile-javac-root.txt [12K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/blanks-eol.txt [15M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/results-checkstyle-root.txt [13M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/results-hadolint.txt [24K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/results-shellcheck.txt [24K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/results-javadoc-javadoc-root.txt [244K] spotbugs: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1469/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn-warnings.html
[jira] [Created] (HADOOP-19037) S3A: S3A: ITestS3AConfiguration failing with region problems
Steve Loughran created HADOOP-19037:
---
Summary: S3A: S3A: ITestS3AConfiguration failing with region
problems
Key: HADOOP-19037
URL: https://issues.apache.org/jira/browse/HADOOP-19037
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.4.0
Reporter: Steve Loughran
After commented out the default region in my ~/.aws/config [default} profile,
test ITestS3AConfiguration. testS3SpecificSignerOverride() fails
{code}
[ERROR]
testS3SpecificSignerOverride(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)
Time elapsed: 0.054 s <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Unable to load region
from any of the providers in the chain
software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@12c626f8:
[software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@ae63559:
Unable to load region from system settings. Region must be specified either via
environment variable (AWS_REGION) or system property (aws.region).,
software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@6e6cfd4c: No
region provided in profile: default,
software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@139147de:
EC2 Metadata is disabled. Unable to retrieve region information from EC2
Metadata service.]
{code}
I'm worried the sdk update has rolled back to the 3.3.x region problems where
well-configured developer setups / ec2 deployments hid problems. certainly we
can see the code is checking these paths
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19036) Standardize Maven Initialization Across Operating Systems
Shilun Fan created HADOOP-19036: --- Summary: Standardize Maven Initialization Across Operating Systems Key: HADOOP-19036 URL: https://issues.apache.org/jira/browse/HADOOP-19036 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.4.0, 3.5.0 Reporter: Shilun Fan Assignee: Shilun Fan The differences in initializing Maven for various operating systems in the build scripts are as follows: - For Ubuntu and Debian, Maven is installed using the yum repository. - For CentOS 7 and CentOS 8, Maven is downloaded remotely. - The Maven version used on Windows is inconsistent with other operating systems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints hadoop.hdfs.TestFileLengthOnClusterRestart hadoop.hdfs.TestDFSInotifyEventInputStream hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.mapreduce.v2.app.TestRuntimeEstimators hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/patch-mvnsite-root.txt [572K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/patch-javadoc-root.txt [36K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [452K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt [44K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1269/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K]
