[jira] [Resolved] (HADOOP-18131) Upgrade maven enforcer plugin and relevant dependencies
[ https://issues.apache.org/jira/browse/HADOOP-18131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma resolved HADOOP-18131. --- Fix Version/s: 3.4.0 Resolution: Fixed > Upgrade maven enforcer plugin and relevant dependencies > --- > > Key: HADOOP-18131 > URL: https://issues.apache.org/jira/browse/HADOOP-18131 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Maven enforcer plugin's latest version 3.0.0 has some noticeable improvements > (e.g. MENFORCER-350, MENFORCER-388, MENFORCER-353) and fixes for us to > incorporate. Besides, some of the relevant enforcer dependencies (e.g. extra > enforcer rules and restrict import enforcer) too have good improvements. > We should upgrade maven enforcer plugin and the relevant dependencies. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.io.compress.snappy.TestSnappyCompressorDecompressor hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.TestRollingUpgrade hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/diff-compile-javac-root.txt [500K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-mvnsite-root.txt [612K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-javadoc-root.txt [76K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [244K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [444K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/594/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-s
[jira] [Created] (HADOOP-18156) Java doc errors in classes like MarkerTool, S3ObjectAttributes etc
Mukund Thakur created HADOOP-18156:
--
Summary: Java doc errors in classes like MarkerTool,
S3ObjectAttributes etc
Key: HADOOP-18156
URL: https://issues.apache.org/jira/browse/HADOOP-18156
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.2
Reporter: Mukund Thakur
{noformat}
home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:856:
warning: empty tag
[ERROR]*
[ERROR] ^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:150:
warning: empty tag
[ERROR]*
[ERROR] ^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:964:
warning: no @param for source
[ERROR] public ScanArgsBuilder withSourceFS(final FileSystem source) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:964:
warning: no @return
[ERROR] public ScanArgsBuilder withSourceFS(final FileSystem source) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:970:
warning: no @param for p
[ERROR] public ScanArgsBuilder withPath(final Path p) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:970:
warning: no @return
[ERROR] public ScanArgsBuilder withPath(final Path p) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:976:
warning: no @param for d
[ERROR] public ScanArgsBuilder withDoPurge(final boolean d) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:976:
warning: no @return
[ERROR] public ScanArgsBuilder withDoPurge(final boolean d) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:982:
warning: no @param for min
[ERROR] public ScanArgsBuilder withMinMarkerCount(final int min) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:982:
warning: no @return
[ERROR] public ScanArgsBuilder withMinMarkerCount(final int min) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:988:
warning: no @param for max
[ERROR] public ScanArgsBuilder withMaxMarkerCount(final int max) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:988:
warning: no @return
[ERROR] public ScanArgsBuilder withMaxMarkerCount(final int max) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:994:
warning: no @param for l
[ERROR] public ScanArgsBuilder withLimit(final int l) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:994:
warning: no @return
[ERROR] public ScanArgsBuilder withLimit(final int l) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:1000:
warning: no @param for b
[ERROR] public ScanArgsBuilder withNonAuth(final boolean b) {
[ERROR]^
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-mul
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/ [Mar 7, 2022 8:01:18 AM] (gyorandr) YARN-11049. MutableConfScheduler is referred as plain String instead of class name. Contributed by Szilard Nemeth [Mar 7, 2022 10:21:10 PM] (noreply) HADOOP-17563. Upgrade BouncyCastle to 1.68 (#3980) -1 overall The following subsystems voted -1: blanks compile golang mvninstall mvnsite pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.yarn.conf.TestYarnConfigurationFields mvninstall: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/patch-mvninstall-root.txt [0] compile: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/patch-compile-root.txt [0] cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/patch-compile-root.txt [0] golang: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/patch-compile-root.txt [0] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/patch-compile-root.txt [0] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/buildtool-patch-checkstyle-root.txt [0] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/patch-mvnsite-root.txt [0] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/patch-javadoc-root.txt [0] spotbugs: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/branch-spotbugs-root.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/branch-spotbugs-hadoop-assemblies.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/branch-spotbugs-hadoop-build-tools.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/branch-spotbugs-hadoop-client-modules.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/branch-spotbugs-hadoop-client-modules_hadoop-client.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/branch-spotbugs-hadoop-client-modules_hadoop-client-api.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/branch-spotbugs-hadoop-client-modules_hadoop-client-check-invariants.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/branch-spotbugs-hadoop-client-modules_hadoop-client-check-test-invariants.txt [0] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/802/artifact/out/branch-spotbugs-hadoop-client-modules_hadoop-client-integration-tests.txt [0] https://ci-hadoop.apache.org/job/hadoo
Re: [ANNOUNCE] Apache Hadoop 3.3.2 release
Thanks Xiaoqiao! yes the change log name is incorrect - I only realized that right before sending the announcement. On Sun, Mar 6, 2022 at 4:10 AM Xiaoqiao He wrote: > Thanks Chao for your quick response and fix. > > It almost works well now except for the changelog link. After checking I > think because of the worry demo of 8.c in [1]. > I have updated wiki[1] and src manually and it seems to work well now. TBH > I didn't dig when the changelog url changed and caused this issue. I am > sure the original guide was working well before 2.10 > > Best Regards, > - He Xiaoqiao > > [1] https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease > > On Sun, Mar 6, 2022 at 10:14 AM Chao Sun wrote: > >> Ah I forgot to do `hugo` before committing the changes. Sorry. It should >> be >> good now. >> >> Chao >> >> On Sat, Mar 5, 2022 at 9:28 AM Chao Sun wrote: >> >> > Hmm interesting. I followed the steps in that second link you posted, >> > and the results looked good when I ran `hugo serve` locally: I can see >> > the announcement and 3.3.2 showed up in the list under the >> > "Documentation" tab. >> > >> > In https://hadoop.apache.org, under the "Documentation" tab, the >> > "Latest" and "Stable" are already updated to 3.3.2, but somehow the >> > announcement is not there. >> > >> > On Sat, Mar 5, 2022 at 1:30 AM Xiaoqiao He wrote: >> > > >> > > Thanks Chao for this great work. >> > > >> > > Just check the hadoop website[1], I didn't get the link or >> announcement >> > > about release 3.3.2. If we missed any steps to publish following >> > > guidelines[2]? Thanks. >> > > >> > > Best Regards, >> > > - He Xiaoqiao >> > > >> > > [1] https://hadoop.apache.org/ >> > > [2] https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease >> > > >> > > >> > > On Fri, Mar 4, 2022 at 1:23 PM Hui Fei wrote: >> > > >> > > > Great work! Thanks, Chao! >> > > > >> > > > Chao Sun 于2022年3月4日周五 09:30写道： >> > > > >> > > > > Hi All, >> > > > > >> > > > > It gives me great pleasure to announce that the Apache Hadoop >> > community >> > > > has >> > > > > voted to release Apache Hadoop 3.3.2. >> > > > > >> > > > > This is the second stable release of Apache Hadoop 3.3 line. It >> > contains >> > > > > 284 bug fixes, improvements and enhancements since 3.3.1. >> > > > > >> > > > > Users are encouraged to read the overview of major changes [1] >> since >> > > > 3.3.1. >> > > > > For details of 284 bug fixes, improvements, and other enhancements >> > since >> > > > > the previous 3.3.1 release, please check release notes [2] and >> > changelog >> > > > > [3]. >> > > > > >> > > > > [1]: https://hadoop.apache.org/docs/r3.3.2/index.html >> > > > > [2]: >> > > > > >> > > > > >> > > > >> > >> http://hadoop.apache.org/docs/r3.3.2/hadoop-project-dist/hadoop-common/release/3.3.2/RELEASENOTES.3.3.2.html >> > > > > [3]: >> > > > > >> > > > > >> > > > >> > >> http://hadoop.apache.org/docs/r3.3.2/hadoop-project-dist/hadoop-common/release/3.3.2/CHANGELOG.3.3.2.html >> > > > > >> > > > > Many thanks to everyone who contributed to the release, and >> everyone >> > in >> > > > the >> > > > > Apache Hadoop community! This release is a direct result of your >> > great >> > > > > contributions. >> > > > > >> > > > > Many thanks to everyone who helped in this release process! >> > > > > >> > > > > Many thanks to Viraj Jasani, Michael Stack, Masatake Iwasaki, >> > Xiaoqiao >> > > > He, >> > > > > Mukund Madhav Thakur, Wei-Chiu Chuang, Steve Loughran, Akira >> Ajisaka >> > and >> > > > > other folks who helped for this release process. >> > > > > >> > > > > Best Regards, >> > > > > Chao >> > > > > >> > > > >> > >> >
[ANNOUNCE] New Hadoop PMC Sun Chao
On behalf of the Apache Hadoop PMC, I am pleased to announce that Sun Chao(sunchao) has accepted the PMC's invitation to become a PMC on the project. We appreciate all of Sun's generous contributions thus far and look forward to his continued involvement. Congratulations and welcome, Sun!
[jira] [Created] (HADOOP-18157) S3AFileSystem cannot find a fake dir any more
JacobZheng created HADOOP-18157:
---
Summary: S3AFileSystem cannot find a fake dir any more
Key: HADOOP-18157
URL: https://issues.apache.org/jira/browse/HADOOP-18157
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.3.1, 3.2.2
Reporter: JacobZheng
I am using spark and delta lake for reading and writing datasets. After I
upgraded the hadoop version, I found that writing files reported an error,
indicating that the folder does not exist.
{code:java}
java.io.FileNotFoundException: No such file or directory:
s3a://table_cache/t5ffe773c987844cbba0aafd/_delta_log
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3356)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3053)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1760)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4263)
at
org.apache.spark.sql.delta.storage.S3SingleDriverLogStore.listFromInternal(S3SingleDriverLogStore.scala:120)
at
org.apache.spark.sql.delta.storage.S3SingleDriverLogStore.exists(S3SingleDriverLogStore.scala:156)
at
org.apache.spark.sql.delta.storage.S3SingleDriverLogStore.write(S3SingleDriverLogStore.scala:174)
at
org.apache.spark.sql.delta.OptimisticTransactionImpl.doCommit(OptimisticTransaction.scala:742)
...
{code}
When this folder has been created, the error is still reported.
The related commit is 49f8ae96, It delete some code that dealing with Fake
directory.
{code:java}
// Either a normal file was not found or the probe was skipped.
// because the key ended in "/" or it was not in the set of probes.
// Look for the dir marker
if (probes.contains(StatusProbeEnum.DirMarker)) {
String newKey = maybeAddTrailingSlash(key);
try {
ObjectMetadata meta = getObjectMetadata(newKey);
if (objectRepresentsDirectory(newKey, meta.getContentLength())) {
LOG.debug("Found file (with /): fake directory");
return new S3AFileStatus(Tristate.TRUE, path, username);
} else {
LOG.warn("Found file (with /): real file? should not happen: {}",
key);
return new S3AFileStatus(meta.getContentLength(),
dateToLong(meta.getLastModified()),
path,
getDefaultBlockSize(path),
username,
meta.getETag(),
meta.getVersionId());
}
} catch (AmazonServiceException e) {
if (e.getStatusCode() != SC_404 || isUnknownBucket(e)) {
throw translateException("getFileStatus", newKey, e);
}
} catch (AmazonClientException e) {
throw translateException("getFileStatus", newKey, e);
}
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [ANNOUNCE] New Hadoop PMC Sun Chao
Congratulations, Sun!👍 Wei-Chiu Chuang 于2022年3月9日周三 11:47写道： > On behalf of the Apache Hadoop PMC, I am pleased to announce that Sun > Chao(sunchao) has accepted the PMC's invitation to become a PMC on > the project. We appreciate all of Sun's generous contributions thus far > and look forward to his continued involvement. > > Congratulations and welcome, Sun! >
[jira] [Created] (HADOOP-18158) Fix failure of create-release script due to releasedocmaker changes in branch-2.10
Masatake Iwasaki created HADOOP-18158: - Summary: Fix failure of create-release script due to releasedocmaker changes in branch-2.10 Key: HADOOP-18158 URL: https://issues.apache.org/jira/browse/HADOOP-18158 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.10.1 Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki The file name generated by releasedocmaker of Yetus was changed from CHANGES.md to CHANGELOG.md. dev-support scripts should be updated along with the change. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [ANNOUNCE] New Hadoop PMC Sun Chao
Congratulations, Sun Chao! Best Regards, - He Xiaoqiao On Wed, Mar 9, 2022 at 1:06 PM tom lee wrote: > Congratulations, Sun!👍 > > Wei-Chiu Chuang 于2022年3月9日周三 11:47写道： > > > On behalf of the Apache Hadoop PMC, I am pleased to announce that Sun > > Chao(sunchao) has accepted the PMC's invitation to become a PMC on > > the project. We appreciate all of Sun's generous contributions thus far > > and look forward to his continued involvement. > > > > Congratulations and welcome, Sun! > > >
